Sessions is temporarily moving to YouTube, check out all our new videos here.

Neos Release Environment / React and Neos

Gerhard Boden speaking at Neos CMS and Flow in November, 2017
Great talks, fired to your inbox 👌
No junk, no spam, just great talks. Unsubscribe any time.

About this talk

Learn about the Neos Release Environment from Gerhard, Neos Core developer.


Okay, so my name is Gerhard. Some of you guys already know me. I've been with the Neos Unicorn team for almost a year now. And I had the great honour to be the release manager of the 3.2 Neos release we released a couple of months ago. And during this time, I learned a lot, how the Neos project is structured and how our release jobs work. And I just want to give you a short insight. So this is not a real fancy talk, so I don't even have any slides, it's just more or less a quick dive into the topic, and I'm also not very well prepared, so don't get your hopes up. So, who has ever used a Neos development collection to do some sort of core contribution? Okay, so what you guys probably already noticed is that the Neos development collection looks quite different from the distribution collection. And the thing is, all those packages that you can see in here are actually just part of the development collection and are split into separate, standalone packages later on. And let's see how this works. So, the first thing, when you want to make a new Neos release, so I am just talking about Neos here, but Bro says it's pretty much the same for Flow, but for consistency and to not confuse everyone I will just talk about Neos. I probably already confused everyone now. So, as you can see here, or probably not, this is a simple git log of the current Neos project. And what you'll probably notice here is that we do a lot of merging from different branches. And this is how the journey always starts. So we take the lowest maintained branch and upmerge it into the next branch, and the next branch, until we end up in master. So, this is always the first step for a new release. This is basically the same if you're doing just a bug fix release, a minor release, or even a major release. There's one tricky thing that we ran into since we did the, vendor namespace change a couple of, I think it's already years now? I think it was last year or something? It was this year. Did we remove it this year? I don't know, maybe it was this year. So this type of free vendor namespace, we took it out, and the problem was that a lot of filenames changed, and even the file structure changed. And this can be quite tricky when you do upmerges, because you end up with the old files that are already stale in this branch, and you suddenly have them in the new one, or even worse, git cannot find the old file, and so it does not have a reference between the new and the old file. And so we had to think about something, and Sebastian, luckily, came up with a solution, because you can also use this in your projects as well if you did some sort of namespace changes. Or you did structural changes to your project. If you put this flag here, this recursive X, and put a rename threshold in it, this will take quite a bit longer, probably, in your project but git will try to look up the files which have been renamed, and when you're lucky, after the merge, everything will, it will give the correct files. So you will not end up with old files in your new branch. And so when you're done, you're hopefully finished with your upmerges, so the next part in your Neos project would be the Jenkins jobs. So you already had to create a branch beforehand. If you forgot that, you would have to start over, probably. Like, we have this job right here, it's called neos-create-branch, and what it basically does, it's just creating a new branch in the development collection. That's not too spectacular, but the really cool thing about this Jenkins jobs is it helps us to actually split this Neos development collection you see here into different standalone packages you can find on packages, like this Neos media package right here, is actually a part of the Neos development collection, but it's automatically separated during the split jobs for you, so you don't have to take care of that. And it also takes care of all the other stuff, like decks, when a new release comes out, because all these packages in here share the same version numbering. So if you look at the Neos media package, it always carries the same tags as all the other core packages. But there are different sort of packages, like optional packages, for example, like the Neos redirect handler, or the Neos redirect packages. You can see here, they don't share the same version numbering because they're standalone packages, so we have to take care of releasing and tagging ourselves. And this can be an advantage, for example, if you want to bring out the bug fix release for this special package, you don't have to wait for the core packages to update, so nobody has to do all this upmerge stuff and make a whole bug fix for Neos and Flow. So you can just go ahead, merge this bug fix, the latest one, and tag a new release, and everybody will receive the Neos redirect handler with the latest bugfixes. So there's ups and downs to both strategies. But we're trying to put normal use stuff into the newest Neos package and newest Flow packages so every new feature and everything that can be split somehow will end up in some separated packages. So this is like our strategy. It probably will never be that modular, like, I don't know, the symphony framework. I don't think there will ever be a time where you can use the Neos media package as a standalone package in every project you want to use. That's not actually a goal, but it just helps us to structure our code and not lose the overview, because you shouldn't put everything into one package. Every programmer knows that. Or everybody's been there. So, well, we talked about these split jobs before, here. So, I just jumped around a little, but what you would actually do if you're done with your upmerges, you would start the Neos Flow release job, and as you can already see here, we have Github webhooks, and they help us to automatically synchronise tags across all packages. So when somebody goes ahead and pushes to the Neos or Flow development collection, this Github webhooks automatically start different jobs. Like, update, this, I already talked about this job, the read-only splits, update references, we don't use this anymore, and the Randa API doc, this is used to read the docs files. So they are already automatically updated as well. And as you can see here, we've got quite a lot of other Jenkins jobs. Like the build minified job that helped us to create minified and Javascript and CSS files until now, this will be replaced in the short future with these two jobs, like the newest build-UI-minified and the newest UI-release jobs. Yeah, and those are probably the most interesting ones here. And after you hit this button, to release, when you're lucky the release job will already take care of everything for you, so every package out on Packagist will always receive the new tag, because it's a completely separate repository on Github. So we never touch these packages, it already says here, read-only. So the split jobs goes ahead and merges and tags the new stuff into here, and afterwards, after quite a few minutes, or in worst case, hours, Packagist will pick up the new tag and everybody who types compose or update on their instance will receive the new packages. Whether it's a new Neos or Flow version or whatever. And, did I forget about something? No, I think that's pretty much the long and short of it. Yeah, I think it's quite interesting to see how this project is structured, because if you just take a quick look and you don't actually notice that there's so many separate packages. And I think it's so cool that we use automated jobs to split it up, but I think that, probably in the near future, I don't know if there are any plans to move away from Jenkins, because there's some sort of, in the community, some sort of discussions if Jenkins is the newest, coolest tools, probably. But yeah, it's pretty cumbersome to write these jobs. So yeah, I don't think that somebody will pick this up. But probably, if one of you guys got interested enough to take care of it. But that's not actually the most important thing to do right now, so yeah. I think that's about it, any question about the Neos release thingies? Can you explain a little bit better the changeflow? Mmhmm, yeah. Good point, good point. I totally forgot to talk about this. Because we automatically create the change logs with every release, so if you ... Ever looked at, let me have a quick look at this package right here ... So with every new release, we always release the Neos change logs, which you can find on Readthedocs, or in the repository itself, of course. Yeah, it's already the fourth restart you can get, so you just type in "readthedocs" and Neos documentation shows up on the first page. I just wanted to show that. Yeah, it's cool! And if you look in here -- Yeah, of course, that's our main focus right now, to stay on top with the Readthedocs keywords. Because everybody knows that Neos' documentation is flawless. Why are you laughing? Is that an official statement? No, it's not. I'm not actually officially here, so I should probably cover this. So you can see here, we have a couple of release notes. And the cool thing about this, this is automatically created, as I already said. Here's another one. It's already a bit outdated, I see, it's from, yeah. This stuff, we write by hand, and the other stuff is automatically merged in from Github, and what I'm trying to find right now is this thing right here. We have some ... No it's not! As I said, I'm awfully unprepared. Let me have a look. Nope, not here either! Because we use these buildscripts, I can show them to you right here. Like create-branch, I already talked about this job, and we have this create-change-log. So these are, I don't think, can you see anything? Can you select the text? Yeah, let's try this. Is it ... No, not really better. Last try, I will open it in a normal text editor so you can see what this is all about. So we automatically create change logs, as I said, four or five times by now. And the coolest thing about this is -- Make it bigger! Yeah, can you read it? It's not that interesting, you don't have to read it, actually, you can just imagine that it's a lot of bad script right here. Yeah, and what it -- Automatically extracted, Github messages and -- Exactly. And you want them to -- Yeah, it looks over all new commits that came with the last release. So basically, when you start a job, you say, this is the last release, and you should look at, like, when it's a minor release, you would say, 3.1.0, if you want to release 3.2.0, and he automatically compares all commits between those two tags, and extracts, like he already said, extracts the commit message, and so you get a clean file with all the commits in there. So the only thing you have to do by hand is to throw out all the bug fixes, for example, if you're doing the release notes for a minor release, because the bug fixes already come with the bug fix releases, and are not interesting for the minor release. So you basically only have to scan through the file and throw out all those lines. But it would be much more cumbersome to do all this by hand. Yeah, this is just an example. Let's have a look at this create branch thingy. Yeah, it just executes a couple of Git commands, and it's just a simple script that runs on Jenkins, but it makes the life for us a lot easier. And we had a server crash, I don't know, it was 1.5 years ago? And all these release scripts were gone because a server in, I think it was Sweden or somewhere? Or Denmark? It just was thrown away. So we had no backups, and they just dumped the server, like literally, dumped the machine. They threw it in the trash, I haven't figured out why the did it. But all this had to be rewritten, so this is actually pretty fresh. Yeah. Any further questions about Neos releases or your Neos problems, or ...