I don't understand why you would want an entire new execution scheduling mechanism just to print results in a specific order. The post dismisses the idea of including an index (a "sequence number" if you really want to be snooty about the idea of counting your requests), but "asynchronously waitable objects" is much, much more mechanism than that.
The irony of all of this is that the core thing the post wants to do, of posting results in order to reduce confusion for users, is also probably (I don't know the complete use case, but in general) wrong! Print when you get the results back! Don't block results behind a slow early request! It's a command line program, what's the virtue of queueing fast responses? The only point of the program is to print the output.
I had to solve a similar problem in Go once, partially to scratch an itch and partially to speed things up.
The S3 AWS SDK has support for doing chunked/parallel downloads if you can supply an io.WriterAt [1] implementation.
type WriterAt interface {
WriteAt(p []byte, off int64) (n int, err error)
}
I created a WriteAtReader that implements io.WriterAt and io.Reader. You can read up to the first byte that hasn't been written to the internal buffers. If there's nothing new to read but there's still pending writes, calls to Read will block and immediately return after the next WriteTo call. As bytes are read through Read, the WriterAtReader frees the memory associated with them.
It provides the benefits/immediate results you get with a streaming implementation and the performance of parallel download streams (at the cost of increased memory usage at the outset).
The first "chunk" can be a little slow, but because the second chunk is almost ready to be read by the time the first is done? zoom zoom.
From the article:
> (The specific context is that I've got a little Go program to do IP to name DNS lookups (it's in Go for reasons), and on the one hand it would be handy to do several DNS lookups in parallel because sometimes they take a while, but on the other hand I want to print the results in command line order because otherwise it gets confusing.)
If I was using this program, I too would appreciate ordered results and an early return where possible without breaking the ordering.
Edit: and other similar programs might use a limited pool of workers, in which case the user would want to see ordered results as they came in and they would be expected to complete roughly in the same order they were started in.
To be clear: you can still easily get this result without anything nearly as complex as promises. But as someone who has written this program many times in many languages, I'm not so sure you're right that this behavior is desirable. People go out of their way to install CLI network tools that don't do this.
I think it's safe to assume that it's a toy problem. A simple example. Picking apart the example is a poor counterargument. It might be difficult to imagine a specific trivial example, but expressing a convincing example concretely can itself be a major diversion. Unless you're saying you can guarantee that your alternative is always better in all conceivable cases?
Similarly, the existence of others wanting different behavior is also a poor counterargument. The fact that someone wants a Bag doesn't mean that someone else is wrong for wanting an Ordered Set.
As you can see upthread, it's not my only argument, just the one responders locked in on. But the problem here is also not lack of an ordered set!
I agree that it seems a bit pointless unless you're running a huge batch and that I don't think reading from channels in a for loop should be referred to as promises, but I've written CLI tools where running batches of jobs in parallel and printing the results in order is desirable. I don't think it's a generally undesirable thing.
Edit: not quite parallel: I mean with a limited pool of workers, or a partial ordering, or something else that means there's a reason to return results early.
Agree RE using sequence/index numbers. I would set up a reply array, send indices along with the requests and store each result in the corresponding index.
Keep track of the lowest received index and print when the next one comes in. Loop the same for the next index if it's already completed.
Or write it in JS where this would be trivial and compile to executable with Deno.
> The only point of the program is to print the output.
A consistent structure helps humans parse the output. I have host(1)'s output order in my mental memory.
Another way to think about this: if the only point of the program is to print the output, why not just serialize, or pipeline only one request past the current one?
Or you could just use a sync.WaitGroup and not print anything until all the results are in, which is what the proposed design does anyways in the case where an early request is the slowest to resolve.
Why not serialize / just print one? Parallel is faster.
Why not wait? Because it’s nice to see results as they come in.
Yes, there are pathological possibilities here, if an early request is slow. But this complaint sounds like a complaint about the author’s goals, which are stated clearly enough. “The author has the wrong desires or goals”, your comment seems to say.
Is it faster? The design proposed deliberately withholds fast results in order to ensure they're printed in sequence. Anybody who has ever run "traceroute" knows what this UX is like. If I wasn't going full TUI, I would just print the results as they come in, on stderr (with an index number).
A sync.WaitGroup is as fast as the proposed behavior, for what it's worth, and doesn't have the ordering complexity.
Dude, just accept the premise of the article.
He gives the requirements: Execute in parallel, print in order as quickly as possible. That’s the premise. The point of the article is in how to achieve that, not whether you should.
“Print as soon as possible and ignore ordering” is not a challenge nor an interesting topic to write an article about.
Yeah, this is funny. It's like presenting someone with a thought experiment and them saying "yeah but that would never happen".
Would a single channel + sequence numbers actually be better, does quantity of channels matter? (I'm genuinely asking, I don't use Go much.)
You can print the results when you get them back but also print them in order too. You’ll need a little bit of ANSI escape magic to do it but it’s not a complicated problem to solve. An example of this done well is docker-compose.
Personally, I’d have written this code using a sequence number as well. Albeit I’d have attached the sequence number to a context. To me, this sound like the kind of problem that contexts do well.
This depends on your definition of “print”. What if your stdout is not a TTY?
Also, if your stdout is a TTY, but your terminal is not ANSI X3.64? You might be able to look up the corresponding escape codes in terminfo based on $TERM (or simply use tput(1)), but the terminal might not even be able to perform those operations. What do you do then?
The ironic thing about the docker-compose UX, which I agree is probably the optimum here, is that it doesn't benefit from promises.
I’m not going to defend the authors desire for promises because it’s a pattern I’ve never liked in any language. But that’s just my personal preferences.
The way I would do it without seeing the implementation:
- Atomic to track print order (name it counter).
- Results slice stores items: {index, err, data}.
- Goroutines: do work, store result, then loop-check if counter matches index to print (with tiny sleep).
Edit: Another solution would be to put printing loop outside with a channel to signal work finished but I don't know how I would avoid using nested loops, one for the items in the list and one to check if the result is back.
So: A channel with buffer size 1, so long as it is only written to once and read from once, feels a lot like a Promise.
A variable you can only read once is a lot less useful than a variable you can read as many times as you want. Same goes for promises.
I don’t think I’ve seen a promise implementation that can’t handle multiple reads after the resolve for many years.
It would not take a lot of glue code to alter the contract to make that happen here.
It's been a while since I did much Go, but I think you can handle this cleanly by making one channel per task, and having an array of channels similar to the array of promises. Each channel takes that task's result, then closes. The caller waits on each channel in sequence.
That's exactly what the author ends up with, too. It's also how you might handle this in any other language that allows for some form of "join", either intended or usable as a join. Waiting on promises, thread joins, channels (as a signal mechanism), it's all the same pattern: Instantiate a bunch of asynchronous activities and queue up a "handler" corresponding to it, wait on the handlers in your desired order.
.NET 9 solves this with
Task.WhenEach
Shortcut could be to create an array of sync.OnceValue, immediately invoking each element in a goroutine. Then iterate through the array and call each function again.
Isn't it even simpler in Go? No channels necessary, Each goroutine gets an index or slice, and writes the results to a shared array.
All you need is the goroutines to report when done via a WaitGroup.
That doesn't satisfy the "report the results as they become available" requirement.
The desired behavior is that printing the first result should only wait for the first result to be available, not for all the results to be available.
Would be trivial to show the results live though, especially for atomic values: https://go.dev/play/p/PRzzO_skWoJ
That's not "live"; that's just polling.
Modified the above to https://go.dev/play/p/DRXyvRHsuAH You get the first result in results[0] thanks to `atomic.Int32`.
package main
import (
"fmt"
"math/rand"
"sync/atomic"
"time"
)
func main() {
args := []int{5, 2, 4, 1, 8}
var indexGen atomic.Int32
indexGen.Store(-1)
results := make([]int, len(args))
finished := make(chan bool)
slowSquare := func(arg int, index int) {
randomMilliseconds := rand.Intn(1000)
blockDuration := time.Duration(randomMilliseconds) * time.Millisecond
fmt.Printf("Squaring %d, Blocking for %d milliseconds...\n", arg, randomMilliseconds)
<-time.After(blockDuration)
idx := indexGen.Add(1)
results[idx] = arg * arg
fmt.Printf("Squared %d: results[%d]=%d\n", arg, idx, results[idx])
finished <- true
}
prettyPrinter := func() {
for range time.NewTicker(time.Second).C {
fmt.Println("Results: ", results)
}
}
go prettyPrinter()
for idx, x := range args {
go slowSquare(x, idx)
}
<-finished
fmt.Println("First Result: ", results[0])
fmt.Println("So-far Results: ", results)
}This is a data race between the write to `results[idx]` in the `slowSquare` goroutine and the read of `results` in the `prettyPrinter` goroutine.
This will wait up to 1 second before showing a result when a result comes in. I'm pretty sure Chris doesn't want any waiting like that.
This will also print some results multiple times. I think Chris wants to print each result once.
It is utterly clear that the random wait is not intrinsic to the logic - it was only added for demonstration to simulate varying duration of requests.
You can simply comment out the println and just pick the first results[0]. Again, the repeated println for all results was only added for demonstrative clarity.
Frankly, the above satisfies all primary goals. The rest is just nitpicking - without a formal specification of the problem one can argue all day.
Lordy, I clicked on this, fully expecting a discussion of a shape-pattern in the board game of Go, and somehow didn't have a thought in my mind for the real, completely different, meaning.
This reminds me of "trompe l'oeil foods"... dishes that appear to be one thing, but are another entirely. Or maybe optical illusions more generally.
Had a similar problem in Elixir - spawn a pool of HTTP requests and notify immediately of the first one that completes with a status of "ok".
Task.Supervisor.async_stream(
My.TaskSupervisor,
my_list_of_urls,
&assess_url_exists/1,
ordered: false,
max_concurrency: 100,
timeout: 30_000
)
|> Enum.find(fn
{:ok, {:ok, _url}} -> true
_ -> false
end)
But the really best part is that any pending computation at the point when the correct result is found, it is automatically cancelled. No waiting for its completion. No memory leaks, no file descriptors left open. It just goes away. Boom.
Just sayin'...
Just sayin'... you have solved a different problem to the one the author describes in the article.
Weird. I thought that "similar" meant "different, but kind of".
The "in-order" requirement makes this a weird problem to think you have. There's no situation where the channels are going to be heavier then the go-routines you're spawning to handle the processing: in fact the last line lamenting leaving blocked go-routines around is weird, because a blocked go-routine is still using less resources then one actually doing things - it's totally fine for them to block waiting to write to the channel because of the "in-order" requirement.
Your worst case scenario is you spawn N go-routines, and they complete 1 by 1 in reverse order from N so your entire dataset is waiting in memory for in-order completion - so no other concern here matters at all.
I love how most of the responses to this article here blame the author for wanting the wrong thing. One could implement a future struct that is awaitable and then the scaffolding to wait on a bunch of them. I have done things like this in other languages, it can be immensely useful. Similarly, when some wanted generics, they were bemoaned for wanting the wrong thing. Go is a wonderful language, even if it is not my cup of tea. But must its users be this defensive?
I don't think this is defensiveness so much as that attempting to model promises in languages like Go will result in bad, nonidiomatic code, and that is a thing that happens semiregularly with Go in particular, as it has so many refugees from other languages with sharply different idioms.
Here, we're discussing implementing future structs just to print DNS results in order. I don't think the UX proposed in the post is good, but stipulate that it is; this is not a hard problem to solve in idiomatic Go.
"refugees from other languages with sharply different idioms"
Reminds me of the time I had to decipher some numerical analysis code written in Common Lisp but in the style of Occam...
> this is not a hard problem to solve in idiomatic Go.
Genuinely asking, what would the solution look like in idiomatic Go?
Let's assume for a second that the premise of the article is valid and exactly the behavior we want - "asynchronous execution but to report the results in order, as each becomes available".
Send the results into a channel, with a separate goroutine that polls the channel, stores a buffer of results, and sorts and prints the latest results at a regular interval.
You could also have N channels, one for each argument, and use reflect.Select to receive the results as soon as available, waiting to print any result until all of its predecessors have come in.
You could also have a mutex-guarded block at the end of every worker goroutine to do the printing and a sync.WaitGroup to follow the workers in main.
> sorts and prints the latest results at a regular interval
Slightly more complicated than that, because you can only sort and print elements once you have _all_ the elements that came before them. Once you add that layer you've got quite a lot more code (and potential for mistakes) than the promises version.
> refugees from other languages with sharply different idioms
Mostly Java or JavaScript devs unable to make a dent in their main programming language space looking for a shortcut to jump to a better paying job by pretending to make a huge difference to the other programming language. I cringe every time a frontend dev who hasn't fully grasped React yet utters the words "I hear Rust is the future. I am going to port a JS package to it..." They have already started making inroads into Golang with half-implemented modules.
Reminds me of how I once wanted to implement a doubly linked list in idiomatic Rust but then people started telling me that I shouldn't want to write that.
How did idiomatic Rust make it hard to implement a doubly-linked list? I don't know Rust, so asking out of curiosity.
Rust's ownership and borrowing rules really want tree-shaped data, i.e. where there are no cycles. But a doubly-linked list is exactly the opposite of that, each pair of adjacent nodes create a cycle.
You can't very easily have two equivalently powerful pointers that point at the same node in Rust. It became somewhat of a meme in Rust community to the point that there is even a series called "Learn Rust With Entirely Too Many Linked Lists" [0].
The memory ownership kinda expects all structures to be in a DAG (ie no cycles since ownership tracking doesn't understand that).
> But must its users be this defensive?
Go is to programing what Brutalism is to Architecture. It is striped bare of everything down to the most basic of forms. It is structure is laid bare, put on display and free of decoration.
If you come in and try to write Golang like other languages your going to be unhappy, your going to tell us how go sucks because NPM/PIP/Gems is better (a common lament). It's not defensive rolling up the news paper and wacking new devs to Golang and telling them "don't do it like its java/c/ruby/js/python do it like this..."
Embrace the less is more of Go and it makes more sense and gets much more pleasant.
Garbage collection screams: "look elsewhere if true brutalism is what you seek; maybe C or Pascal deserve that mantle more than a high level language like Go."
I don't think that is a fair comparison for Brutalism. Programming's brutalism is more like Scheme: Simple, composable, sound and principled design. Golang is a "modern" McMansion: it tries to copy a modern style by unintelligently stripping away all ornamentation, but it doesn't understand the underlying theme of modernism and ultimately fails to achieve a coherent design.
This is the HN thread I didn't know I needed until I saw it.
I often wonder how many of Brutalism's ardent fans grew up in an environment dense with Brutalist structures. I'm not here to bash the movement (the finest examples are wonderful) but just a reminder that the median Brutalist building is often a bleak, bleak affair.
There's probably an analogy in there somewhere for programming too. Perhaps the best of Golang is wonderful but the median just doesn't compare.
ETA: Either way, thanks for the taking the discussion down this direction. Certainly made me chuckle.
it is possible they fear being replaced with a few lines of erlang