11.4
The Operations, and What collect Really Does
Thirty methods split cleanly in two. Does it hand back a stream, or an answer. Once you sort them by that question, the list stops being a list to memorise.
Previously on
Section 11.3 gave you the shape: a source, some intermediate operations, one terminal operation. It also gave you the rule that matters, which is that intermediate operations do nothing until a terminal one arrives.
You have three methods. filter, map, toList. There are about thirty, and they all obey that same rule.
The problem
The list of stream methods looks like something to memorise. It is not, and treating it that way makes it much harder than it is.
Here are the names, unsorted:
filter map flatMap distinct sorted peek limit skip takeWhile dropWhile
forEach toList collect count reduce min max findFirst findAny
anyMatch allMatch noneMatch toArray iteratorTwenty four names with no obvious order. Learning them one at a time is slow and it does not stick.
They split in two, and the split is mechanical. Look at what each one gives back. About half return a Stream, and half return something else. That single fact decides everything about how the method behaves, and you can read it off a signature without remembering anything.
The other problem is collect. You have used toList, which is the easy case. Then you meet this:
staff.stream().collect(Collectors.groupingBy(Emp::dept, Collectors.counting()));Two nested calls to a class you have not met, one of which is passed as an argument to the other. It is the piece of stream code that looks least like Java, and it is also the one that does the most work.
The idea
Sort them by return type and the list organises itself.
| Intermediate. Returns a Stream, does nothing yet | Terminal. Returns an answer, runs everything | |
|---|---|---|
| Pick some | filter, distinct, limit, skip, takeWhile, dropWhile | findFirst, findAny |
| Change each | map, flatMap | toList, toArray, collect |
| Put in order | sorted | min, max |
| Ask a question | anyMatch, allMatch, noneMatch, count | |
| Squash to one | reduce | |
| Do something | peek | forEach |
Two pairs in that table are worth looking at side by side.
peek and forEach do the same thing to each element. peek is intermediate and forEach is terminal, so peek alone runs nothing at all. It exists for looking, not for doing.
sorted and min both need an order. sorted gives you back a stream and does no comparing yet. min gives you the answer and does all of it.
Three of these lean on work you did in earlier phases.
sorted() with no argument needs Comparable, from Section 10.8. Without it you get a ClassCastException at run time. sorted(Comparator) takes the order from outside instead, which is the whole point of Comparator.
distinct() asks equals, from Section 8.3. Miss the override and it does nothing, silently.
min and max return Optional, because an empty stream has no smallest element.
Now collect. The word to hold on to is recipe.
List<String> names = staff.stream().map(Emp::name).collect(Collectors.toList());collect says: gather these up. Collectors.toList() is a recipe telling it how. Collectors is a class full of ready made recipes, and that is all it is.
Collectors.toList() // into a List
Collectors.toSet() // into a Set, duplicates dropped by equals
Collectors.joining(", ") // into one String with a separator
Collectors.counting() // how many
Collectors.averagingInt(f) // the average of a number pulled out of each
Collectors.toMap(k, v) // into a Map, given how to get keys and valuesThe one that earns its keep is groupingBy:
Map<String, List<Emp>> byDept = staff.stream()
.collect(Collectors.groupingBy(Emp::dept));One line. You get a Map whose keys are the departments and whose values are lists of the people in each. Writing that by hand is a loop, a HashMap, and a computeIfAbsent call, which is exactly what you wrote in Section 10.6.
And recipes nest. The second argument says what to do with each group instead of just listing them:
Map<String, Long> countPerDept = staff.stream()
.collect(Collectors.groupingBy(Emp::dept, Collectors.counting()));
// {Sales=2, Eng=2}That is the nesting that looked alarming a moment ago. Read it as: group by department, and for each group, count. The second recipe describes the group.
reduce squashes a stream to one value.
staff.stream().map(Emp::salary).reduce(0, Integer::sum); // 300Start at 0. Take the running total and the next salary, add them, keep going. Every other terminal operation that produces one value is a special case of this.
Under the hood
Going deepergroupingBy gives you a HashMap, and it shows. Run the grouping on staff added in the order Aditya, Rohit, Sonu, Rohan:
{Sales=[Sonu, Rohan], Eng=[Aditya, Rohit]}Sales comes first, and Eng was added first. There is no bug here. groupingBy builds a HashMap, and a HashMap has no order, which you proved for yourself in Section 10.5. If you want insertion order, ask for it:
Collectors.groupingBy(Emp::dept, LinkedHashMap::new, Collectors.toList())distinct fails silently on your own classes. Two objects, same value, no equals override:
List<NoEq> raw = List.of(new NoEq(1), new NoEq(1));
raw.stream().distinct().toList(); // [N1, N1]Both survive. distinct used Object.equals, which compares references, and two separate objects are two separate references. No exception is raised and nothing warns you.
Compare with the boxed integers, where equals exists:
Stream.of(1, 2, 2, 3, 1).distinct().toList(); // [1, 2, 3]Same method, opposite outcome, and the only difference is a method on your class from Phase VIII.
reduce has two forms, and the types tell you why.
reduce(0, Integer::sum) // returns int, 300
reduce(Integer::sum) // returns Optional[300]With a starting value, an empty stream still has an answer, so an int can always be returned. Without one, an empty stream has no answer to give. Java will not invent zero, because you never said zero was the identity, so the return type carries the possibility of nothing instead.
That is the same reason min, max, findFirst and findAny all return Optional.
peek runs only for elements that get that far.
staff.stream().peek(e -> print(e.name())).limit(2).toList();peek Aditya
peek RohitTwo of four. peek is inside the pipeline, so it obeys the same one element at a time rule as everything else. It is not a logging statement that runs over your collection. It is a stage, and stages downstream can stop it early.
With no terminal operation at all, peek prints nothing, which is the Section 11.3 rule showing up in the place people most expect an exception to it.
flatMap is map for when each element becomes many.
Stream.of(List.of(1, 2), List.of(3, 4))
.flatMap(List::stream)
.toList(); // [1, 2, 3, 4]map would have given you a stream of two lists. flatMap opens each one and pours the contents into a single stream. Any time you have a collection of collections and want one flat run of values, this is the operation.
What it costs
Some of these operations quietly stop being lazy.
sorted is the clearest case. Nothing can be known about the smallest element until every element has been seen, so sorted holds the whole stream in memory before it emits anything. Put it on an endless stream and the program never finishes. The same goes for distinct, which has to remember everything it has already seen.
Order of operations turns into a performance decision because of that. filter before map means the map runs on fewer elements. limit before an expensive map means it runs ten times instead of a million. The answer is identical either way, so nothing tells you when you have written the slow version.
Collectors is also a large surface. There are around forty recipes, some of which nest inside others, and reading an unfamiliar one takes real effort. groupingBy with three arguments is genuinely hard to follow the first time.
There is a smaller trap in toMap that is worth knowing before it happens. Two elements producing the same key throws IllegalStateException: Duplicate key, which is arguably the right call and still a surprise. The three argument form lets you say what to do when they collide.
And peek misleads people constantly. It looks like a place to log. It runs only for elements that reach it, only when a terminal operation exists, and in a parallel stream it runs on whatever thread happened to be there. As a debugging tool it is fine. As a way to make something happen, it is the wrong method.
Check yourself
Answer from memory before opening the explanation. Then compare your rule with the answer and revisit the example if they differ.
How do you tell an intermediate operation from a terminal one without looking anything up?
Show the answer
Look at what it returns.
An intermediate operation returns a
Stream. That is what lets you keep adding dots, and it is why it cannot have done any work: it has nothing to report yet.A terminal operation returns anything else. A
List, a number, aboolean, anOptional, or nothing at all in the case offorEach. The chain stops there because there is no stream left to add to.So the return type is the rule, not a category you have to remember. If you can keep typing a dot and get more stream methods, nothing has run yet.
`distinct()` on a list of your own objects leaves the duplicates in. Why?
Show the answer
Because
distinctasksequals, and your class did not override it.Without an override,
equalsis the one inherited fromObject, which compares references. Two objects holding identical values are different references, so both are kept. This is the same failure as Section 8.3, arriving somewhere new.It fails quietly. No exception, no warning, just a list that still has duplicates in it. Same for
Collectors.toSet,groupingBykeys, and anything else that has to decide whether two things are the same.Fix the class, not the stream. Override
equalsandhashCodetogether, as the contract in Section 8.3 requires.`reduce(0, Integer::sum)` gives you an `int`. `reduce(Integer::sum)` gives you an `Optional<Integer>`. Why the difference?
Show the answer
Because of the empty stream.
With a starting value, an empty stream has an obvious answer: the starting value. Sum nothing and you get 0, so there is always something to return.
Without one there is no answer at all. What is the sum of no numbers, when you were not told where to start? Not zero, because you never said zero. There is nothing honest to return, so Java returns an
Optionalthat may be empty.The type is telling you about a case you might not have thought about, which is the whole argument for
Optionalin Section 11.5.
Exercises
Write each solution in VS Code, predict its result, and then run the checks. Reading an example does not replace implementing it.
4 exercises100 pointsabout 100 minutes
Sort Them Yourself
ex-11-4-aDo the sorting before you check the answers. That is the exercise.
You should be able to place all twenty without knowing what any of them do, purely from what they hand back. If you find yourself reaching for what a method means, you are doing it the hard way.
The last two questions are the point. Two pairs of methods that sound like the same job land in different columns, and in both cases the return type explains it completely.
What your program must do
- Split all twenty into two columns using only the return type
- Check every one of them against what your editor reports
- Explain why peek and forEach land in different columns
- Explain why sorted and min land in different columns
import java.util.*;
import java.util.stream.*;
public class SplitThem {
public static void main(String[] args) {
// Put each of these in one of two columns, without looking anything up.
// Use only this question: does it hand back a Stream, or an answer?
//
// filter map flatMap distinct sorted peek limit skip
// forEach toList collect count reduce min max findFirst
// anyMatch allMatch noneMatch toArray
// TODO: write your two lists in a comment
// TODO: then check each one by typing it and reading what your editor says
// TODO: peek and forEach do the same thing. Which column is each in, and why?
// TODO: sorted and min both need an order. Which one does the comparing?
}
}
Hint 1
Stream, you can keep typing a dot and get more stream methods. That is the test, and it also tells you nothing has run yet.Hint 2
peek is intermediate, so on its own it does nothing at all. forEach is terminal, so it runs the whole pipeline.Hint 3almost the answer
sorted hands back a stream and has done no comparing yet. min hands back an answer, so it has done all of it. The return type told you which.The Duplicates That Would Not Go
ex-11-4-bPredict all three before running anything, and write the numbers down.
Every one of these is asking the same question underneath, and every one gets the same wrong answer. What makes it worth doing is that nothing goes wrong in a way you could notice. No exception. No warning. A list that still has duplicates in it and a set that is somehow the same size as the list.
The fix is not in the stream. Go back to the class and add the two methods together, because adding one is its own bug.
What your program must do
- Predict the result of distinct, toSet and groupingBy before running any of them
- Run all three and record what actually happened
- Add equals and hashCode together, then run all three again
- Say why none of the three failures produced an error message
import java.util.*;
import java.util.stream.*;
public class StillThere {
static class Unit {
final String name; final int id;
Unit(String n, int i) { name = n; id = i; }
public String toString() { return name + "#" + id; }
// no equals, no hashCode. On purpose.
}
public static void main(String[] args) {
List<Unit> units = List.of(
new Unit("Atlas", 101), new Unit("Atlas", 101), new Unit("Beacon", 102));
// TODO: distinct(). Predict the size first. Then run it.
// TODO: Collectors.toSet(). Predict, then run.
// TODO: groupingBy(u -> u). Predict how many keys, then run.
// TODO: now add equals and hashCode, and run all three again.
}
}
Hint 1
equals is the one inherited from Object, and that compares references. Two separately created objects are never the same reference.Hint 2
Hint 3almost the answer
HashSet finds the bucket with hashCode and only then asks equals, so an override of equals alone never gets consulted.One Line Instead of Fifteen
ex-11-4-cWrite the loop version first. It matters that you do this in that order.
The loop is about six lines and you have written it before, in Section 10.6, with computeIfAbsent. Once it is on the screen, groupingBy stops being a new idea and becomes the same six lines with a name.
The last part is a small trap worth walking into. Print the map and the departments come out in an order you did not choose. Work out why before reading the hint, because you already proved this to yourself back in Phase X.
What your program must do
- Write the grouping by hand first, with a loop and computeIfAbsent
- Replace it with groupingBy and confirm the answers match
- Count per department and average salary per department, using nested collectors
- Explain the order the map comes out in, then change it
import java.util.*;
import java.util.stream.*;
public class Grouping {
record Emp(String name, String dept, int salary) { }
public static void main(String[] args) {
List<Emp> staff = List.of(
new Emp("Aditya", "Eng", 90),
new Emp("Rohit", "Eng", 70),
new Emp("Sonu", "Sales", 60),
new Emp("Rohan", "Sales", 80));
// TODO: group by department, BY HAND, with a loop and computeIfAbsent
// TODO: the same thing with Collectors.groupingBy. Compare the two.
// TODO: count per department, using a second collector inside the first
// TODO: average salary per department
// TODO: print the map. Is Eng first? Why not? Then make it first.
}
}
Hint 1
map.computeIfAbsent(e.dept(), k -> new ArrayList<>()).add(e) inside a loop.Hint 2
groupingBy says what to do with each group instead of listing it. Collectors.counting() and Collectors.averagingInt(Emp::salary) both go there.Hint 3almost the answer
groupingBy builds a HashMap, and a HashMap has none. The three argument form takes LinkedHashMap::new in the middle and fixes it.Where You Put limit Changes Everything
ex-11-4-dSame chain, same answer, wildly different amount of work.
Put a counter in the map and move limit around it. The result never changes. The number of times your code ran changes by a factor of a hundred thousand, and there is nothing in the output to tell you which version you shipped.
The last part is a different kind of lesson. Most stream operations pass elements through one at a time, and two of them cannot. Work out why sorted is forced to break the rule, and distinct will explain itself.
What your program must do
- Count map calls with limit before and after the map, and confirm the answers match
- Do the same for filter before and after map
- Say which order you would write by default and why
- Explain why sorted on an endless stream never finishes
import java.util.*;
import java.util.stream.*;
public class Placement {
static int mapCalls = 0;
static String expensive(String s) {
mapCalls++;
return s.toUpperCase();
}
public static void main(String[] args) {
List<String> names = new ArrayList<>();
for (int i = 0; i < 1_000_000; i++) names.add("name" + i);
// TODO: map then limit(10). Count the map calls.
// TODO: limit(10) then map. Count again. Same answer?
// TODO: filter then map, and map then filter. Count both.
// TODO: add sorted() to a chain over an endless stream. Predict what happens
// BEFORE you run it, and be ready to stop the program.
}
}
Hint 1
Hint 2
Hint 3almost the answer
sorted cannot emit anything until it has seen everything, because the smallest element could always be the next one. On an endless source there is no everything, so it waits forever. distinct has the same problem for the same reason.After the credits
Four operations in this section refused to promise you a value.
staff.stream().min(bySalary); // Optional<Emp>
staff.stream().findFirst(); // Optional<Emp>
staff.stream().reduce(Integer::sum); // Optional<Integer>Every one of them can be asked about an empty stream, and none of them will return null to you.
Section 11.5 is about the type that replaced it. Optional is not a null check with extra steps, though it turns into one if you use it wrong. It is a return type that makes the empty case impossible to forget, and the wrong way to use it looks like this:
if (result.isPresent()) { ... } // you have reinvented the null checkThe same section covers orElseGet, which takes a Supplier from Section 11.2 so that an expensive fallback is only built when it is actually needed.
Collector will return in 11.6 - Parallel Streams and the Primitive Ones