This is a technique advocated by Leslie Lamport. It will take quite a bit of work to get right.
It looks a little silly to have a 1 at the very bottom :-).
TeX will run out of memory if you take this too far.
If somebody wants to look at page 1 and then page 50, they have to wait for the whole dvi file to download.
See project 5, Scrolling within a page to show linked-to text.
A different approach which does not have this problem involves writing a new parser, based on idvi.display.PageParser. This parser would ignore the kTokenBeginPage and kTokenEndPage tokens, creating a single large Block structure for the whole document. This would allow cross-page links, colored text, and outlines.