vitiral 2020-12-28 04:25 UTC link

I'm a huge fan of the ideas behind this library. Using the stable ABI of "magic numbers" is brilliant. Would love to see this solution gain more usage!

mmastrac 2020-12-28 04:35 UTC link

It appears to build a "naked" thompson shell script without a shebang line that works as a PE file for Windows/DOS, and uses `exec` to feed a binary to the shell via a pipe.

Interesting approach for a universal binary.

qlk1123 2020-12-28 04:36 UTC link

Sorry for being naive, but how can it be possible that nobody had thought of this approach and this approach would have actually worked? What have been preventing people from coming up with something like this?

RcouF1uZ4gsC 2020-12-28 04:43 UTC link

> For an example of how that works, consider the following common fact about C which that's often overlooked. External function calls such as the following: memcpy(foo, bar, n); Are roughly equivalent to the following assembly, which leads compilers to assume that most cpu state is clobbered:

I think most modern C and C++ compilers know that memcpy is special and optimize accordingly. I believe some C++ compilers recognize when you are using memcpy just to do bit casting to different representations and avoid doing a memory copy at all.

jws 2020-12-28 05:11 UTC link

Doesn’t the “strlcpy” example near the bottom of the page overwrite the buffer by one byte when the source string is as long or longer than the buffer?

Imagine I have a three byte buffer, n is 3 and I copy in “foo”. The result of the MIN is going to be three. And d[3] is the fourth byte of my three byte buffer.

1vuio0pswjnm7 2020-12-28 06:11 UTC link

Here's the story behind this:

https://justine.storage.googleapis.com/ape.html

groodt 2020-12-28 06:33 UTC link

This is very clever. I wonder if it could be adopted for Python C extensions (where possible) to make it easier to distribute cross-platform binary distributions. Of course it wouldn’t be suitable for gui toolkits, but I imagine that the majority of Python C-extensions could be handled.

BeeOnRope 2020-12-28 06:46 UTC link

This approach of using inline asm wrappers around functions which specify exactly the clobbered registers is neat, but...

I don't think the approach of making a call from inline asm is safe. In particular, it clobbers the redzone [1], which the compiler may have used and expect to be intact after the call.

In general, at least some compilers (e.g., gcc) assume there are no calls in an inline asm block.

---

[1] https://godbolt.org/z/e4ecKr

erik_seaberg 2020-12-28 06:46 UTC link

In case it helps anyone else, this is a compact libc for an x86_64 binary format that can bootstrap and make syscalls on several kernels but relies on static linking (it doesn’t use any platform’s shared libc).

smcameron 2020-12-28 06:49 UTC link

> Please note this is intended for people who don't care about desktop GUIs, and just want stdio and sockets [...]

Not to diminish the achievement at all, but a significant limitation to be noted (and not a very surprising limitation).

I also wonder about things like the sizeof(long). On 64-bit linux, this is 8, but on windows, it's 4. Well, "on windows"... maybe this throws all the APIs that make such assumptions, uh, out the window, (puts on sunglasses).

TimTheTinker 2020-12-28 07:04 UTC link

This is an interesting, and potentially compelling argument that WebAssembly binaries and runtime actually aren’t the future of compile-once-run-anywhere systems software: these support syscalls, sockets, and stdin/stdout —- all of which requires platform-specific glue code to achieve using WebAssembly and standard libc (or JS).

parhamn 2020-12-28 07:27 UTC link

In theory the linked APE library [1] can also be used for languages like Rust and Go binaries to make fat portable ones as well, right?

[1] https://justine.storage.googleapis.com/ape.html

rav 2020-12-28 09:04 UTC link

I have Wine installed on my Linux system, which uses "binfmt_misc" to automatically run Windows-looking executable files through Wine. This causes an issue with Cosmopolitan, since it uses an output format that triggers binfmt_misc. I had to run this to disable binfmt_misc on my system:

    sudo systemctl mask proc-sys-fs-binfmt_misc.automount

...and this to disable it for just the current boot:

    sudo bash -c 'echo 0 > /proc/sys/fs/binfmt_misc/status'

pwdisswordfish5 2020-12-28 09:37 UTC link

Cute trick, but probably fragile. It basically requires either writing functions like memcpy in assembly by hand or parsing the compiler-generated assembly post-codegen to determine the actual set of register clobbers. Otherwise it is bound to be broken as soon as the register allocator decides to clobber a different set of registers when generating the function.

Also, this effectively makes the body of the function a part of its public ABI, which makes it morally equivalent to inlining; it requires users of the function to be recompiled each time the body (and in consequence the clobber set) is changed. But since this seems to be static-linking-only library, this seems not that much of a problem in this case.

Still, I’m expecting improvements to whole-program LTO and inter-procedural optimisations to render this hack obsolete.

mk89 2020-12-28 10:29 UTC link

This is one of the most interesting projects I have seen this year. Congratulations for the idea and for the execution.

etamponi 2020-12-28 10:48 UTC link

This project sounds amazing and very interesting! I have one question: how does it work on bare metal? I tried to look for an explanation in the documentation but I couldn't find anything. The specific questions I have are:

1) How does this interact with a boot loader like GRUB? Or does it come with its own boot loader? But in this case, shouldn't this go in a specific location on the boot disk?

2) How does it handle io in bare metal mode? Specifically: sockets.

mintyc 2020-12-28 11:53 UTC link

A couple more links that may be helpful in this pursuit:

https://sagargv.blogspot.com/2014/09/on-building-portable-li... and

https://blog.ksub.org/bytes/2016/07/23/ld.so-glibcs-dynanic-...

Watch out for the implications of licenses etc.

Uptrenda 2020-12-28 12:12 UTC link

Now all we need is a good package manager for it and you're looking at a C renaissance

fwsgonzo 2020-12-28 12:12 UTC link

This is what I've been working on with RISC-V, and you have no idea how much resistance I get when I try to fix the problems relating to calling conventions, inline assembly and system calls.

I don't want to have to explain what I'm doing from the beginning every time I have a complex question about how to make a function call from inline assembly. It has really turned me off from stackoverflow as a whole. Sorry for the rant.

I think your project is awesome, and really shows that we can improve the base upon which everything rests.

ndesaulniers 2020-12-29 18:37 UTC link

> So what Cosmopolitan does for memcpy() and many other frequently-called core library leaf functions, is defining a simple macro wrapper, which tells the compiler the correct subset of the abi that's actually needed

By not marking all memory as clobbered for a call to memcpy, couldn't the memcpy overwrite a local variable, in which case reads post memcpy could miss the updated value?

> The above two compiles used the same gcc flags

Why does the before have frame pointers and post does not?

mmastrac 2020-12-28 04:51 UTC link

I think it's been pondered a bunch, but the legwork required to build the headers for each system so that you can execute the same binaries on every platform was tough.

Sebb767 2020-12-28 04:53 UTC link

I can think of two reasons: It's arguably a hack and it's not needed.

While this is awesome, it can easily break (zsh already seems to do so) and, as the author states, it is not meant for GUI apps or programs relying deeply on OS APIs, which includes a lot of apps.

Additionaly, it does not solve an important problem. Doing a build for each supported OS is not that hard and easily automated. The actual problem is to be able to reuse code and this is solved for the most part.

CyberDildonics 2020-12-28 04:57 UTC link

Even if you are trying to avoid the standard library, visual studio will insert memcpy into a loop that is just copying memory, then the linker will error out when it can't find it.

choppaface 2020-12-28 05:01 UTC link

Not an answer, but Splunk currently has a large deployment of a C++ application that supports a large variety of platforms. At least as of a few years ago, they deployed C++ indexing and querying nodes on customer machines. They've spent a great deal of time to accommodate the peculiarities of enterprise customers, old compilers, etc. One might imagine that a "build once run anywhere" platform would benefit them, but they've found a business model where they were able to work around the peculiarities and still be profitable.

So while there's probably some interesting historical reasons that explain why there's no "JVM for C++", Splunk is a good example of a modern technical / business model where conquering platform fragmentation was feasible.

jart 2020-12-28 05:01 UTC link

Author here. Here's the best explanation I've thought of so far:

> If it's this easy, why has no one done this before? The best answer I can tell is it requires an minor ABI change, where C preprocessor macros relating to system interfaces need to be symbolic. This is barely an issue, except in cases like switch(errno){case EINVAL:...}. https://justine.lol/ape.html

It also took an act of willpower. This is a novel approach to building software that says the conventional wisdom is wrong. When I first stumbled upon the idea, I couldn't believe it was even possible, until I actually did it. Now I'm glad that I did. Actually Portable Executable has been so useful for my work. I used to work on the TensorFlow team, so I've seen the worst c/c++ build toil imaginable, and looking back at those experiences I just can't believe how I managed to get by for so long without having portable build-once binaries without interpreters.

modeless 2020-12-28 05:15 UTC link

A downside of this approach is the binary can't be executed with the exec system call because the kernel doesn't recognize scripts without a shebang. It can only be executed from within a shell.

comex 2020-12-28 05:25 UTC link

Seems that way to me as well.

harikb 2020-12-28 05:26 UTC link

I think you are right. `strlcpy` is only supposed to copy `n-1` bytes from src to dest.

https://github.com/freebsd/freebsd/blob/master/sys/libkern/s...

jart 2020-12-28 05:48 UTC link

Author here. The example and repository has been updated. Many eyes make all bugs shallow. I was hoping to do more testing before posting on Hacker News. Now that the cat's out of the bag, I'd like to explain what's being done to make sure bugs like that never happen.

Cosmopolitan implements a simplified version of the Address Sanitizer (ASAN) runtime. See https://github.com/jart/cosmopolitan/blob/master/libc/log/as... I've been systematically working towards using it to vet the entire codebase. It's helped to address more than a few memory issues so far. However more work needs to be done polishing the ASAN code so it can be bootstrapped for the purposes of having memory safety in the lowest level libraries like LIBC_STR, due to the way that the Makefile requires acyclicity between libraries.

This has been less of an issue, because bugs are normally very evident in functions like strcpy(). The BSD version is an exception though, since it was only used to port programs from the OpenBSD codebase which were used sparingly. But in either case, the real focus here is having continual incremental improvements over time, and I'm very committed to addressing any and all concerns.

Cosmopolitan also has extensive unit tests. This is something that C libraries have traditional done somewhat sparingly, if at all, since C library authors conventionally rely on an ecosystem of ported software that very quickly breaks if everything isn't perfect. If you build the Cosmopolitan repository, you'll notice there's 194 independent test executables in the o/test folder. That's one of the great things about Actually Portable Executable. It makes binaries so tiny that we can have lots of tests, completely isolated from each other!

jart 2020-12-28 06:58 UTC link

Cosmopolitan doesn't use the "red zone" because these binaries boot on bare metal and operating system code can't assume a red zone. GCC supports this. Otherwise the Linux Kernel wouldn't work, since it's compiled the same way. So invoking call from inline asm is perfectly safe. The only real issue is that the called function can't assume the stack is 16-byte aligned. But that's fine, since Cosmo only using asm(call) to call functions written in assembly.

jart 2020-12-28 07:03 UTC link

Cosmopolitan would actually be a great fit for Python extensions, since manylinux1 requires RHEL5 support and Cosmopolitan is probably the only modern C library that still supports it (glibc these days needs rhel8+). Maybe someone will figure out a way to compile a single-file library that's simultaneously a .dylib / .dll / .so just like how Actually Portable Executable does for programs.

jart 2020-12-28 07:21 UTC link

Author here. WebAssembly has played an important role protecting CI on the Internet and it's cool that it enables us to run LLVM in the browser. I was very surprised when I learned that some people were using it offline outside of browsers, since doing so requires a JVM-like runtime such as wasmtime. Cosmopolitan proves that it's possible to just fix C instead, which might fix all the stuff that's built on top of C too. We can in fact have first-class portable native binaries, which will only require JIT compilation if they're launched on a non-x86 machine. This is probably how Java would have been designed, if Intel's instruction set hadn't been encumbered by patents at the time. That's changed. The x86-64 patents just expired this year. So I say why not use it as a canonical software encoding.

jart 2020-12-28 07:33 UTC link

Yes! Cosmopolitan and APE are now permissively licensed under ISC terms. So they'd be a perfect fit for language authors. One of the biggest obstacles language authors encounter, is overcoming the obstacles posed by portability between operating system apis. Cosmopolitan takes care of that. You can just use the normal textbook unix apis and have it work on Windows/Linux/BSDs. Best of all, Cosmpolitan is tiny, embeddable, and unobtrusive. As in, these "fat binaries" are actually more like 16kb in size. So any language author who chooses to use it is going to have more time to focus on their vision, while reaching a broader audience from day one, with a sexy lightweight form.

rurban 2020-12-28 07:33 UTC link

Replace modern with clang, and awful with gcc. I have no idea why gcc is still so fast overall, but its memcpy is horrible, as shown in the blog post. My own memcpy in my libc also beats the gcc/glibc one. The compilers are only good for short memcpy's. For the rest the asm hinders modern compiler optimizations.

jart 2020-12-28 07:50 UTC link

Author here. That's not an inherent limitation. See https://justine.lol/apelife/index.html for an example of a GUI + TUI that's built using Cosmopolitan. The reason why I said what you quoted, is I'm simply trying to calibrate expectations. I don't view desktop GUIs as a productive area of focus, because there's such a lack of consensus surrounding the APIs that requires, and web browsers do a great job.

As for sizeof(long) it's 64-bit with Cosmopolitan, which uses an LP64 data model. That hasn't impacted its ability to use the WIN32 API. See https://github.com/jart/cosmopolitan/tree/master/libc/nt where WIN32 API functions are declared using uint32_t, int64_t, etc. It works great. I compile my Windows apps on Linux and it feels wonderful to not have to touch MSVC. Cosmopolitan even takes care of the chore of translating your UTF-8 strings into UTF-16 when polyfilling functions like open() which delegates to CreateFile() on Windows.

jart 2020-12-28 09:16 UTC link

Give this a try:

    sudo sh -c "echo ':APE:M::MZqFpD::/bin/sh:' >/proc/sys/fs/binfmt_misc/register"

That command should allow Actually Portable Executables to co-exist with WINE, which assumes the MZ prefix, since APE binaries always use the longer MZqFpD prefix.

saagarjha 2020-12-28 11:07 UTC link

Sadly, the "magic numbers" are much less stable than you'd think.

jart 2020-12-28 11:24 UTC link

Author here. Bare metal so far has been a "we built it because we could" kind of feature, so please be warned it's not as polished. Here's a quick tutorial.

Basically, ape/ape.S is a traditional BIOS bootloader. It also embeds a GRUB stub, which basically just switches back to the real mode BIOS bootloader entrypoint. So it should work with or without GRUB. The boot sector loads the rest of your executable of disk and prints "actually portable executable" to your display. It then configures standard i/o to use serial uart, sets up the page tables, enters long mode, and calls _start(). That's all I've got working so far. I would love to have e1000 and virtio though!

To try it out, here's how you build the included PC BIOS system emulator and boot an x86-64 ape executable that prints a spinning deathstar:

    m=tiny; make -j12 MODE=$m o/$m/tool/build/blinkenlights.com o/$m/tool/viz/deathstar.com &&
      o/$m/tool/build/blinkenlights.com -rt o/$m/tool/viz/deathstar.com

Video screencast: https://justine.lol/cosmopolitan/deathstar.html

See also: https://justine.lol/blinkenlights/manual.html

Enjoy!

oh_sigh 2020-12-28 12:43 UTC link

This is such a nicer compliment than it will be in a week.

kenniskrag 2020-12-28 13:19 UTC link

I reached the same conclusion about stackoverflow.

Is there an incentive from the real world to answer questions because so many beginner try it?

xyproto 2020-12-28 13:21 UTC link

I have the same experience. Stack Overflow is not great when you try to achive something novel or do something new from scratch.

orra 2020-12-28 14:07 UTC link

It's beautiful, isn't it? Making something simultaneously compatible with multiple formats. A far more interesting use of the PE/COM “MZ” header than simply printing “This program cannot be run in DOS mode”.

nn3 2020-12-28 17:38 UTC link

WebAsm is not performance competitive:

"We find that the mean slowdown of WebAssembly vs. native across SPEC benchmarks is 1.55× for Chrome and 1.45× for Firefox, with peak slowdowns of 2.5× in Chrome and 2.08× in Firefox."

https://arxiv.org/pdf/1901.09056.pdf

timClicks 2020-12-28 21:06 UTC link

These binaries wouldn't provide the isolation that wasm provides... Although it's looks like they're essentially unikernels, which means that they could potentially be run per-invocation by a hypervisor.

ahh 2021-01-07 03:48 UTC link

I can confirm we came up with this optimization inside tcmalloc, and our compiler experts nacked it as unsafe, even only calling tightly controlled code. A pity, as it is really beautiful to see the improved clobbering (and even, in some cases, hacked calling conventions.)

Editorial Channel

What the content says

Structural Channel

What the site does