Alpha opening selections

Hello everyone!

It’s been a week since the application phase ended, and now it’s time to announce the results.

First of all, thanks to everyone who submitted an application. There were even more applications submitted than last time, and after discounting duplicates, the total number is 616 applications, including everyone who applied last time. You keep outperforming my expectations 😛 I’m glad that the community shows no signs of dissolving or diminished interest even though it’s now 7 years since the game closed. I’m confident the day will come when everyone will be able to play the full game again, and I’m glad to know the community will still be around.

This opening has also shown some weaknesses in the current application processing infrastructure. There was no easy way to double-check whether your application was still recorded if you had submitted one last time, and also no easy way to check that your application was submitted properly if you submitted it in this opening. The integration with website accounts, discord, and the LU server itself also isn’t ideal at the moment. For future applications, I’d like to develop an interface where you can submit an application and check on its status later on, maybe even as an interface of the LU server itself.

Because of this uncertainty in whether your application had been processed, some users submitted additional applications under the same name. Of course, these won’t disqualify you from the alpha, and we’ve also taken steps to ensure everyone is only counted once.

I’ve taken the time and read through every application that was submitted, and they’ve been a great read, with many of you putting a lot of effort into detailing your memories of this game. Thank you all for sharing these nostalgic moments.

There were a lot of wonderful applications, and it’s sad we can’t select all. I really hope we’ll be able to do another opening soon, so that you’ll have another chance at getting to play the game. However, today, let’s take a look at the selections for the current alpha opening.

We’re randomly selecting 20 players out of 616 applications, which means if you entered you have a 3.24% chance of being selected.

The selected player for the alpha opening are:

  • ArieegeeCZ
  • Caminon
  • Darklair
  • Drakensin
  • FrostEST
  • JWGrieves
  • LegoFan222
  • MAYBUTNYE
  • Masterkdt
  • ShockPain
  • Smurdy
  • TacklessSix5
  • TheLegorza
  • Turtlegaming5
  • Wheathan
  • WhovianForever
  • andrew
  • fffffplayer1
  • gamera2001
  • iSetless

Congratulations to all of the above. The selections are quite varied, some applied this opening, some applied last time; some were FTP players, some were members, and some never got the chance to play.

If you’re one of the selected, check your mail, info on how to connect has been sent to your mail address. (Check your spam folder too, just in case.)
If you’re not one of the selected you may still have a chance in later selections, there will be a number of openings in the future.

I’m glad to welcome 20 new testers to the server. Every opening brings us one step closer to having the full game working again.

See you all in-game!
– lcdr

New alpha opening

Hey everyone!

With the new network layer working, the time has finally come for another opening in the current alpha. It’s been one and a half years now since the start of the alpha, and a lot of things have changed since then:

  • The alpha provided some useful real-world experience of how the server responds to a number of players being connected at the same time. I initially only scheduled the server to run on weekends, but as the server ran better than expected, I switched the server to run 24/7.
  • The alpha testers have found and reported more than 100 issues, and a lot of them have been fixed already. There are still a bunch of issues remaining, but with the network layer rewrite functionally complete and other rewrites planned, the project is moving closer to fixing them.
  • The work on the MLN server has helped to both get a revival of a related game going, and to get some insight in how to design the rewrites needed to address the most problematic bugs found during the alpha.
  • The first of the rewrites has now completed the minimum viable implementation, and work is under way to make the new network layer work entirely under the hood.

The rewrite of the network layer has at the same time removed a large blocker for a new alpha opening, and introduced a new component that would benefit a lot from real-world testing. Testers have previously experienced connection issues that would result in their connection getting dropped, especially during world loads. The network layer rewrite attempts to fix that, but connection issues are notorious for only happening to some specific users in specific situations, so having more testers would help in making sure the issue is resolved.

A new opening in the alpha is also a good opportunity to keep testing active, as the activity naturally slows down a bit over time.

With these factors coming together, it’s time for a new opening up the alpha to new testers. The server development is still very much focused on bug fixing and getting the parts that are implemented to work like in the original. This is my top priority because there’s not much point to features if they don’t work right. That’s why this new stage doesn’t mark the introduction of new features to the server. However, there has been progress in server development since the initial alpha release, which is why we’ve decided to launch a new stage. To avoid misunderstandings, we’ve chosen to call it “opening in the current alpha”, as opposed to “new alpha”, and we’ll be using the latter only for releases with new features.

With all that out of the way, it’s time for the specifics of the new opening. We’ve chosen to keep much of the process we used for the initial alpha, as it’s worked quite well.

  • The new alpha opening will start on April 26th. On this day the new testers will be able to join the server and play.
  • There will be 20 tester slots available for this opening. Twenty new people will double the count of testers with access to the server.
  • People wishing to be testers will need to apply to the alpha with a short description of why they want to be a tester. This system will help to ensure that the selected are dedicated, and will discourage duplicate/spam submissions.
  • If you had applied to the initial alpha, you will be counted automatically, without having to resubmit an application. Don’t worry, we haven’t forgotten about you.
  • Those eligible to be selected (having written a valid application for any alpha opening) will be selected randomly by a computer. This will ensure that everyone has equal chances of getting in.
  • The period in which you can submit an application starts now, and ends two weeks from now, on April 9th.
  • We’ll be looking through the applications the week after that, and announce the results on April 16th. The selected testers will receive a mail with instructions and will be able to access the alpha tester channel on the project discord.
  • Finally, on April 26th, the selected testers will be able to join the server and play.

We’ve written up some more detailed explanations in our Alpha FAQ.
If you want to apply for this alpha opening, click the link below.

Apply for Alpha

It’s been one and a half years now since the initial alpha opening, and I’m glad we’re now finally at the point where we can let more people play. The server still has ways to go until it will be complete, but it’s slowly and steadily progressing, and I’m confident we’ll get there one day. With the new alpha opening, we’re a step further along.

See you all in-game!
– lcdr

New network layer

When I was working on the MLN server, one thing that made development much easier is that the lowest levels were automatically taken care of, as much of the work had already been done by the TCP and HTTP protocols. In contrast, my work on the LU server has led to me having to start at the bottom and implement the RakNet protocol, which takes the role of TCP in LU’s networking. My experience with the MLN server got me thinking that perhaps it was possible to redesign LU’s networking so that the benefits of MLN’s networking could also be applied to LU. This is what led me to develop a new network layer that I’ve now finished work on.

The protocols involved here are situated at the transport layer of networking. At this layer, there are two protocols that are widely in use: UDP and TCP. These protocols are involved in basically every traffic on the internet. UDP is an extremely simple protocol, and essentially all it does is providing ports to associate network packets with a specific process on a computer. At this layer, network traffic is still very unreliable; packets sent from one computer may get lost in the network and never arrive at the destination. They may also arrive in the wrong order, which is also important for many applications. UDP does not attempt to solve these problems, and therefore packets sent using the UDP protocol are unreliable. TCP however implements special mechanisms under the hood to ensure that packets will reliably arrive, and will arrive in the proper order. A lot of applications (websites, email, etc) need packets to be reliable, but there are some for which unreliable packets are actually useful, like DNS. However, for these applications, this is usually a clear choice, they’ll either go full unreliable or full reliable.

Games however pose a special case in that they sometimes need reliable packets, and sometimes need unreliable ones. Here are some examples from LU:

  • Logging in to the game needs to be a reliable packet, because it’s important that the server receives the login request – if not, you’d be stuck at the login screen forever.
  • The position updates sent by the client when the player walks around don’t actually need to reach the server 100% of the time – they are sent so frequently that it doesn’t matter if one is lost. Therefore position updates are more suited to unreliable packets.

Therefore LU can’t just decide on one of UDP or TCP and stick with it, it needs features from both. This is what the RakNet protocol tries to solve: it allows LU to choose between unreliable and reliable transmission for every packet. RakNet works by using UDP as a base, and then builds mechanisms on top that ensure reliability. But unlike TCP, these mechanisms are optional, and can be enabled per packet.

This seems like a solid solution for this problem. However, RakNet has a few quirks in its implementation that make using it less than optimal:

  • In addition to the distinction between “reliable” and “unreliable”, RakNet also supports some more fine grained options that are somewhere between raw UDP and a full TCP implementation. However, at least in LU’s case, these special modes are almost never used – it’s usually all or nothing.
  • RakNet has some flaws in its protocol design that make it unnecessarily complicated, the same could be achieved more elegantly.
  • RakNet is a niche protocol, with far less support than UDP or TCP. This makes it more likely to have bugs and inefficient implementations.
  • LU’s copy of the RakNet implementation is statically compiled into the client .exe, which means it can’t be updated, especially since LU itself is no longer updated.
  • RakNet implements its own combination of cryptography to encrypt its connection, and the encryption isn’t enforced. Together with the fact that RakNet is niche and more than 10 years old at this point, the protocol cannot be said to be secure.

Thus, there’s room for improvement for LU’s networking situation. While working on the MLN server, I had an idea for a possible solution: The same kind of per-packet distinction could be achieved by using both UDP and TCP simultaneously, sending on UDP when unreliable packets were needed, and on TCP when reliable packets were needed. The solution seemed both simple and obvious enough that it was more a surprise that LU wasn’t using it already.

However, even though this seems straightforward in theory, LU’s situation made it quite complicated to actually implement in practice. We can’t just switch LU from one protocol to another, since RakNet is baked into the program and can’t be removed. Therefore, what I’ve been working on the past few months is a “shim”: A program running on the client side, acting as a local server for the RakNet protocol, translating the traffic to the TCP/UDP based protocol, and relaying it to the actual server. Due to the way LU works, this has been a bit more complicated than initially thought, as the shim also needs to simulate multiple server instances for when the player switches worlds. However, after a lot of development and testing, I’ve been able to get it fully working.

With the TCP/UDP protocol fully working, it was straightforward to swap out the TCP protocol for the encrypted TLS protocol, which is widely used, for example to secure HTTPS connections. With the protocol now using this cryptographic industry standard, LU’s security is now rock solid.

Additionally, the UDP and TCP implementations are provided by the operating system, so they can be improved without the program having to change. This also means that they are optimized to the full extent possible, as the same implementations are used for things like high speed file transfer and video streaming.

I’ve done a few unscientific benchmarks, loading worlds in LU with the old RakNet protocol and the new TCP/UDP protocol, and there are some quite noticeable improvements:

Values are time from when the client signals clientside load completion to the end of the loading screen, on localhost, without encryption, in seconds:

WorldOld TimeNew Time
Venture Explorer42
Crux Prime272
Avant Gardens363

Edit: It seems my connection or my ISP’s firewall may have significantly slowed down RakNet’s loading times in this test. It appears RakNet can be significantly faster on better connections, but the new protocol still seems to outperform it there.

Previously, loading a world could be quite slow, mostly because the congestion control and retransmission timeout algorithms used in my RakNet implementation seemed to have some problems estimating the ideal values for the amount of packets per second to send, and how long to wait until the packets would be resent. With the new implementation, everything is lightning fast, and the complexity is outsourced to the TCP implementation of the operating system.

Next steps

However, it’s possible to make these improvements even better. To keep backwards compatibility with the RakNet protocol, right now the shim needs to run in the background all the time as a visible console application. The client’s boot.cfg also needs to be changed to to configure the client to connect to the shim instead of connecting to the server directly. This isn’t as user-friendly as I’d like it to be. The complexity of running the shim and translating between the protocols also adds some overhead, though it’s usually small compared to network latency.

However, these things can be fixed. As I’ve mentioned above, the RakNet implementation is baked into the client .exe – it’s not possible to replace it, as you’d need to be able to recompile the client. This isn’t 100% true – there is a way of modifying the client to run code instead of RakNet’s network functions: .dll hooking. Dll hooking works by getting the client to load a dynamically linked library at startup. The code in this library then has access to the internals of the client process, and with some surgical precision, it can find the location of the machine instructions of the RakNet functions and hijack them to run its own code instead. This technique has been successfully done in a number of games to mod them without source code. However it’s also much more intricate and complicated than the shim solution, which is why I chose to focus on completing the shim first to show that this kind of protocol replacement is possible and valuable. Now that that’s done, I can further look into a .dll-hooking-based solution, which should in theory be able to provide record speeds at less overhead.

In the meantime, I’ll be releasing both the shim executable and source code in the next days, so that other projects are also able to make use of this new protocol.

Alpha

The network layer has been very important to me in making a decision about opening the ongoing Alpha test to new players. Some alpha testers have previously reported getting randomly disconnected from the server sometimes, which this new protocol should fix. Along with the faster loading times and the increased security, this new protocol is vital in improving the user experience of the testers. Therefore it has been an absolute requirement to me for a new alpha opening. With the protocol completed, the outlook for an alpha opening now looks much better… but more on that in the next progress report 😉

See you all in-game!
– lcdr

Alpha selections

Hello everyone!

It’s been a week since the application phase ended, and now it’s finally time to announce the results.

First of all, thanks to everyone who submitted an application. In total, we received 303 applications, more than I’d ever have expected. I’m really impressed with how many people are still around who want to play LU, even though the game closed 5 years ago, and how many would apply even though this is just a first alpha. I went ahead and read every one of those 303 applications you sent us, which took a couple of hours, but I felt it was the least I could do to honor your dedication to this game. Your applications were much more than I imagined, with many of you detailing fun moments in original LU, your enthusiasm for the game, and your excitement for being able to play again. It was a great and inspiring read, thanks to you all for taking the time to write all this.

The application system also seems to have worked well to prevent spam; out of 303 applications only 1 was spam (and don’t worry, we didn’t accidentally classify your application as spam, the application in question was a clear impersonation we manually confirmed before disqualifying it).

Now, before continuing with the selections, a note on how the system works: You submitted your application in the application phase. From these 302 applications we selected 20 to be included in the first alpha stage starting on October 26th. The selections were made randomly, using a computer (to be precise, a small python script). However, these 20 people are just the first stage. To evaluate the server performance under load, we’re starting with a small amount, and will add more people in later stages. This also means that if you submitted an application and didn’t get selected, you still have a chance, because your application also counts for the later stages. If you haven’t had a chance to submit your application, you will also still have a chance in later stages, where we will be accepting new applications as well.

Now, on to the selections!

We’re selecting 20 players out of 302 applications, which means if you entered you have a 6.62% chance of being selected.

The selected players for Stage 1, starting on October 26th, are:

 

  • Komplexus
  • Dazoo1
  • ethanlindley
  • Tinka
  • Baffish
  • lordpickleking
  • Homie0Jr
  • emanresU
  • Liam3997
  • Astrophel
  • mater06
  • Mickeystick
  • Halofan4566
  • Matthew
  • LS2112
  • Raydiation
  • DocGrognard
  • creeperkid75
  • stefanoss
  • ShiWham

Congratulations to all of the above. I’m glad the random algorithm provides for so much diversity. We’ve got all kinds of LU fans in there, from original beta testers to live players, to FTP players, to people who never got the chance to play, people from the developer side and people who are engaged in the community, young ones and AFOLs who played LU with their kids. I’m really looking forward to all of them coming together in-game to play LU again.

If you’re one of the selected, check your mail, we will provide you with info on how to connect later today.
If you’re not selected you may still have a chance in later selections, as mentioned above.

And no matter whether you’re selected or not, we will continue to share info about our progress and the alpha with you. You’ll be able to look forward to alpha gameplay videos not only from our youtube, but from the players as well, as we allow streaming and recording of the alpha.

Remember, this all starts on October 26th! I’ve been looking forward to this next step towards everyone being able to play again for years now, and I’m excited that it’s only 5 more days until then.

See you all in-game!
– lcdr

 

 

Alpha release date & timeline

Hello everyone!

We’ve done a lot of server testing over the last month. I’m glad to say that we’ve been able to fix the bugs that came up, and implement the necessary server administration features. We believe we’re ready for a first alpha, where we will invite people from the community to test our server.

The first alpha stage will begin on October 26th, 2017.

We’ve mentioned before that we’d like the alpha selection process to work in a fair way, and we’ve come up with a process that we think will help with this.

We’ve decided not to base alpha access on giving out keys, but instead via applications. Everyone who wants to join the alpha can submit an application describing why they want to join, and after filtering out spam we randomly select from the submissions. This will help with duplicates, and complications with people asking for keys.

We’ve decided to start low with 20 slots available for the first alpha stage. This is so we can analyze how well the server responds to multiple players playing simultaneously, and possibly optimize the server to work more efficiently under these conditions. After we complete our analysis, and implement bug fixes and optimizations, we will expand the alpha in later stages to include more players and features.

To explain the process, here’s a timeline:

September 30 (Today) Alpha application phase starts. From now on, people are able to submit their application for alpha.
October 14 Alpha application phase ends. People will still be able to submit applications, but they will count towards a later stage, not the one beginning on October 26.
October 21 After filtering out spam and randomly selecting applications, we announce the selections on October 21. Selected applicants will be notified via mail about the next steps.
October 26 Alpha Stage 1 starts, with 20 selected people.
Later Once testing with the initial group of 20 people is successful, we will open up the alpha to more people in later stages.

We’ve written up some more detailed explanations in our Alpha FAQ. Check it out if you’ve got a question, or post in the forums.

You’ll be able to submit your applications to the first stage starting now, until October 14. Click below to start your application for a chance to be selected.

Apply for Alpha

We’ve been wanting to do an alpha for a while now, and we’re glad we’re now able to start the process. Check back on October 21 when we announce the selections.

See you all in game!
– lcdr