!= View Source: Twitter.com - Sessions by Pusher

View Source: Twitter.com

Giuseppe Gurgone speaking at viennaJS in January, 2017
Great talks, fired to your inbox 👌
No junk, no spam, just great talks. Unsubscribe any time.

About this talk

An overview of twitter.com's front-end architecture.


This is News Source Twitter. So hello, everyone. Today, I'm going to give you a very brief overview of Twitter's dotcom architecture. My name is Guiseppe. I'm from Italy. Born and raised in Sicily, actually, but for the past seven years, I've been living around the world. Louder? Is that okay? I have a website that I rarely update that is giuseppe.pizza, and the fact that I'm Italian and I like pizza just by coincidence. I'm Guiseppe Gurgone, and I joined Twitter as a user in 2009. I'm not related whatsoever to twitter.com, so please don't then sue me tonight. Okay, when I joined Twitter, this is how it looked like. It was very web 1.0 kind of thing. I believe it was server rendered with a bit of Javascript on top of it. It was still pretty slick for that time, but a little over one year later, they built...they release a new UI completely, redesigned from scratch. And they had this side bar on the right. I believe a contextual side bar that will use for many things like show you details about the tree, or like the slide before, information about the profile and so on. They called it !NewTwitter, and that's how they sort of promoted it. I put a bang there, because around that time is when they introduced the famous hashbang, using your fragment to do... basically routing navigation between pages and all that. It was a full client-side application, a very complex one, I believe. Even with a very minimum amount of pages, was still doing very interesting stuff, and invested a lot in open source, both contributing and using it using JQuery mustache like JS [inaudible], and all this kind of fancy stuff at that time. And also, everything was built using an home-grown MVC. I believe that Backbone at that time was probably quite young, and whatever. Like many other companies, they figured they'd build their own. And as they were adding new features and all this kind of stuff, obviously, things got more complex and all that, but it was necessary because it was big, single page application, basically. Yet, two years later, and bam, they did it again. They revamped their UI, and now, the side bar is gone. And tweet became expandable, virtually, basically. So now you could expand a tweet, and the timeline would expand and show you information. And around that time also, they introduced those previews. They had some partners like YouTube and all those kind of services and showing previews, as you expand every tweet. And it was also around that time that I was using Twitter a lot and had this idea of what if Twitter had inline previews? I was kind of pissed to click every time to see what was the picture about, the YouTube video about, and eventually, I ended up building a browser extension called Previeweet that would do exactly that, adding a small image on the right side of each tweet. And that's when my journey to twitter.com search got started, to reverse engineer the shit out of it. It was really literally having minified code on the side, like, beautified, and then rewriting the relevant part on my editor like there. And obviously, I learned a lot of stuff. For instance, why use true when you can use not zero, or the tilde operator, and comma, and lots of interesting things around minifiers and all this kind of stuff? And well, eventually I made it. I built this browser extension, that's been open source from day one, and this is the last iteration before they release Twitter cards. It looks familiar, probably, but that was before Twitter cards. Anyway, so around that time, with this new release, they sort of changed things a little bit. They went back to service rendering, reducing the time to first tweet of one fifth, and they went with progressing announcement. They built a JavaScript framework, very small JavaScript framework, that was component-based, was at this point, I think React wasn't even open source. And all the alternative were pretty much MVC-like stuff. They were writing their code in CommonJS, and then wrapping them in AMD module that I would... Loadrunner, that is pretty much the same thing, very similar to RequireJS. They also, at this point, they said, "Hi," to, basically, to the famous hashbang in February of PushState and a more reliable Europe. So we're going to go through those small main pieces, obviously. I'm not able to cover all of them, but most of the important things are right there. We're going to look at Flight.js, their JavaScript framework, that eventually they open sourced in 2013, how they do code splitting, or how they organize bundles, how they initialize the app and finally routing. So Flight.js, it's basically a component-based framework and it is event-driven. Components are totally isolated from each other and they communicate. They rather express intent by firing event and something else in the page can so describe [inaudible] and then react in some way. And that means that you could remove one component and the app still works. And when they released this open source, I was like, "Yo, this is it." I mean, React wasn't even open source, so probably it was about the same times. It was very good. Flight is more like a framework to build frameworks, in a way, because it doesn't come with a router or state management libraries and those kind of stuff. You have to build them yourself. But in a nutshell, it's just a constructor function. Optionally, you can define some default attributes that are similar to React props. And then you can have some logic around functions and first in software, the component is initialized. It's something similar to component [inaudible] of React, if you're familiar with that, where you can register event handlers and do your stuff. But basically, component just fire events, they can manipulate the DOM and so on. You eventually would define your component with flight.component and passing in this function, and then using it by attaching to a DOM node. You can either pass in a select or just an existing DOM node, and you will attach that component to the DOM and it will start to work. So every Flight components get referenced to the DOM node that it's attached to. You have two versions, a JQuery one and a plain DOM node. And Twitter specifically organized their components in UI and Data components. In reality, there is no difference... there's not such a...this classification is not a thing in Flight, it's just their own decision. They decided to add VR components to manage user interaction, update the DOM and all this kind of stuff, while data component just communicated with the server, fetched data and so on. They're completely decoupled, and as I say, they can trigger and subscribe to DOM event. They're total independent, basically, and that makes them very easy to test, because what you have to test, it's just this contract, basically. You have to test the firing event or did they react to some event? I had the chance to build an application with Flight, and it was super easy and also back-end developer were super able to write JavaScript without any problem. And most importantly, they're very easy to reason about, because now you just have to understand a small component and probably this tweet summarizes perfectly. "If the poo is trapped in a box, it'll only smell when you open it." That means that you could put whatever you want inside your component, but it was very isolated and you could refactor it later and no problem. Didn't matter how bad it was, the code, and it would still work and it wouldn't affect the entire application architecture. So moving onto code splitting, if you open the source stub in the developer's tools of any twitter.com page, you will see this. And basically, they have two main bundles. One is init and the other one is common. And then there is a page specific bundle, so every page has their own bundle, gets their own bundle, and those bundles look like this, pretty much. They're just a JavaScript module, they export a simple function, calling it, where they attach a lot of those components to the page. That's it. If you inspect any page, then you will find these four guys, and there are three needing input, and one script tag. The first input has an ID init-data, and it contains a huge JSON block that contains application data state and all those kind of stuff. It's similar to... I don't know if you've ever done server-side rendering with Redux, and you have an initial state, and you dump in a global object, or whatever. Then they have another hidden input, where they store the module that they want to load for this page, in addition to init and common, and then the base path for the module loader, that is with this swift-module-path. And eventually, they require the init bundle. In the hidden bundle, there's a lot of things, but the most important, in my opinion, is this one that is a tiny boot loader that does four main things. The first is to grab those initial data, like the JSON that you saw here, and then grab the module path that the module loader is going to need to find every bundle. And then grab all the modules with class swift-boot-module, in this case it was have pages permalink. And finally, just requiring using these like, down there is like require of RequireJS. That comes from Loadrunner. There is a module other than Web Road. They basically require all those modules and then they...which, if you remember, every module would export this function, and they just call this function with the initial data. And that's how they boot pages and...until a few days ago, I was preparing my slides, and this guy tweeted that they're moving to webpack. I took a quick look, and seems like it's still an ongoing process. They're definitely moving to webpack to bundle the JavaScript in all...every assets, actually. Well, for that matter, probably they will move to React in the future. I don't know. Finally, routing, they used PushState via the HTML5 History API. They rely a lot on Progressive Enhancement. So URLs works, even... Usually, if you have JavaScript disabled, your page will allow it, because they render server-side. And by here, they have the same distinction of UI components and data. They have two components to manage routing. One is a UI component that does nothing but listen for clicks on the JS-nav links. Whenever a link happens, he just fires a custom event. UI navigate with the information, usually the href of that link. And somewhere else, there is a data component who is listening to that...listening to that event and fetches the data and then updates the...if everything goes well, the updates the URL and history and fires a custom event, again saying, "You know, everything went well. Here's the data and here's the payload. Whoever's interested can have data page." And obviously, there's this guy again, the UI component, that will grab those data and update the UI. There's a great talk by Todd Kloots that is called <i>PushState to the Future.</i> He did a great job by explaining how all this works. You definitely might want to check it out. But that's what's [inaudible]. Thank you. Yeah, if you have question, obviously, there's a lot more.