{"id":61,"date":"2008-10-12T17:50:11","date_gmt":"2008-10-12T16:50:11","guid":{"rendered":"http:\/\/jimblackler.com\/blog\/?p=61"},"modified":"2009-02-04T11:29:33","modified_gmt":"2009-02-04T10:29:33","slug":"bringing-yield-return-from-c-to-java","status":"publish","type":"post","link":"https:\/\/jimblackler.com\/?p=61","title":{"rendered":"Bringing &#8216;yield return&#8217; from C# to Java"},"content":{"rendered":"<p>Consider a function that collects and returns a list of results. It might look like this:<\/p>\n<div id=\"ig-sh-1\" class=\"syntax_hilite\">\n\n\t\n\t<div class=\"code\">\n\t\t<ol class=\"java\" style=\"font-family:monospace\"><li><div style=\"font: normal normal 1em\/1.2em monospace;margin:0;padding:0;background:none;vertical-align:top\"><span style=\"color: #000000;font-weight: bold\">public<\/span> ArrayList<span style=\"color: #339933\">&lt;<\/span>String<span style=\"color: #339933\">&gt;<\/span> retrieveAll<span style=\"color: #009900\">&#040;<\/span><span style=\"color: #000066;font-weight: bold\">long<\/span> requesterId<span style=\"color: #009900\">&#041;<\/span><span style=\"color: #339933\">;<\/span><\/div><\/li>\n<\/ol>\t<\/div>\n\n<\/div>\n\n<p>Or even better, this, so it could use anything that implements Iterable&lt;&gt;:\n<\/p>\n<div id=\"ig-sh-2\" class=\"syntax_hilite\">\n\n\t\n\t<div class=\"code\">\n\t\t<ol class=\"java\" style=\"font-family:monospace\"><li><div style=\"font: normal normal 1em\/1.2em monospace;margin:0;padding:0;background:none;vertical-align:top\"><span style=\"color: #000000;font-weight: bold\">public<\/span> Iterable<span style=\"color: #339933\">&lt;<\/span>String<span style=\"color: #339933\">&gt;<\/span> retrieveAll<span style=\"color: #009900\">&#040;<\/span><span style=\"color: #000066;font-weight: bold\">long<\/span> requesterId<span style=\"color: #009900\">&#041;<\/span><span style=\"color: #339933\">;<\/span><\/div><\/li>\n<\/ol>\t<\/div>\n\n<\/div>\n\n<p>However, storing the results in a container which is then returned may be inefficient for the application. It may be better to enable the calling code to act immediately on each collected result rather than first waiting for all results to be collected and stored. If that were possible..<\/p>\n<ul>\n<li>No memory would be used to store a list.<\/li>\n<li>Results could be presented to the user straight away.<\/li>\n<li>Calling code could abort the collecting process part way through based on its own logic &#8211; for instance if it already has more results than it can handle.<\/li>\n<\/ul>\n<h3>Custom iterators<\/h3>\n<p>One way to avoid an intermediate list is for the collecting function to construct a custom Iterable&lt;&gt; object, which constructs Iterator&lt;&gt; objects that contain the collecting logic. Each execution of next() calculates the next value and returns it straight away for the calling code.<\/p>\n<p>The difficulty for the programmer is that this may require a significantly different structure of the collecting code. This code has to store any state from result to result as private variables inside the iterator. Call stack is reset with each result, so algorithms that use recursion are not possible.<\/p>\n<p>More convenient for the programmer would be a technique that allows the collecting code to keep control of the machine state during the collecting process.<\/p>\n<h3>Yield return<\/h3>\n<p>C# has &#8216;yield return&#8217; which allows a function to be structured exactly as if it is building a list. It may keep its own state in local variables and the call stack, but still present each value to the calling code immediately as it is found.<\/p>\n<p>Java doesn&#8217;t have yield return, but another pattern that could be used is this:<\/p>\n<div id=\"ig-sh-3\" class=\"syntax_hilite\">\n\n\t\n\t<div class=\"code\">\n\t\t<ol class=\"java\" style=\"font-family:monospace\"><li><div style=\"font: normal normal 1em\/1.2em monospace;margin:0;padding:0;background:none;vertical-align:top\"><span style=\"color: #000000;font-weight: bold\">public<\/span> <span style=\"color: #000066;font-weight: bold\">void<\/span> retrieveAll<span style=\"color: #009900\">&#040;<\/span><span style=\"color: #000066;font-weight: bold\">long<\/span> requesterId, ResultProcessor<span style=\"color: #339933\">&lt;<\/span>String<span style=\"color: #339933\">&gt;<\/span> collector<span style=\"color: #009900\">&#041;<\/span><span style=\"color: #339933\">;<\/span><\/div><\/li>\n<\/ol>\t<\/div>\n\n<\/div>\n\n<p>Where <b>ResultHandler<\/b>&lt;&gt; is a simple interface to an object invoked by the collecting code to return individual results as they are collected.<\/p>\n<div id=\"ig-sh-4\" class=\"syntax_hilite\">\n\n\t\n\t<div class=\"code\">\n\t\t<ol class=\"java\" style=\"font-family:monospace\"><li><div style=\"font: normal normal 1em\/1.2em monospace;margin:0;padding:0;background:none;vertical-align:top\"><span style=\"color: #000000;font-weight: bold\">public<\/span> <span style=\"color: #000000;font-weight: bold\">interface<\/span> ResultHandler<span style=\"color: #339933\">&lt;<\/span>T<span style=\"color: #339933\">&gt;<\/span> <span style=\"color: #009900\">&#123;<\/span><\/div><\/li>\n<li><div style=\"font: normal normal 1em\/1.2em monospace;margin:0;padding:0;background:none;vertical-align:top\">&nbsp; <span style=\"color: #000066;font-weight: bold\">void<\/span> handleResult<span style=\"color: #009900\">&#040;<\/span>T value<span style=\"color: #009900\">&#041;<\/span> <span style=\"color: #000000;font-weight: bold\">throws<\/span> CollectionAbortedException<span style=\"color: #339933\">;<\/span><\/div><\/li>\n<li><div style=\"font: normal normal 1em\/1.2em monospace;margin:0;padding:0;background:none;vertical-align:top\"><span style=\"color: #009900\">&#125;<\/span><\/div><\/li>\n<\/ol>\t<\/div>\n\n<\/div>\n\n<p>It is left up to the calling code what ResultHandler&lt;&gt; implementing object to supply, and how to handle the data. Anonymous classes would be quite neat here. For instance, a caller could output the results to a console like so:<\/p>\n<div id=\"ig-sh-5\" class=\"syntax_hilite\">\n\n\t\n\t<div class=\"code\">\n\t\t<ol class=\"java\" style=\"font-family:monospace\"><li><div style=\"font: normal normal 1em\/1.2em monospace;margin:0;padding:0;background:none;vertical-align:top\">retreiveAll<span style=\"color: #009900\">&#040;<\/span>myId, <span style=\"color: #000000;font-weight: bold\">new<\/span> ResultHandler<span style=\"color: #339933\">&lt;<\/span>String<span style=\"color: #339933\">&gt;<\/span><span style=\"color: #009900\">&#040;<\/span><span style=\"color: #009900\">&#041;<\/span><span style=\"color: #009900\">&#123;<\/span><\/div><\/li>\n<li><div style=\"font: normal normal 1em\/1.2em monospace;margin:0;padding:0;background:none;vertical-align:top\">&nbsp; <span style=\"color: #000000;font-weight: bold\">public<\/span> <span style=\"color: #000066;font-weight: bold\">void<\/span> handleResult<span style=\"color: #009900\">&#040;<\/span><span style=\"color: #003399\">String<\/span> value<span style=\"color: #009900\">&#041;<\/span> <span style=\"color: #009900\">&#123;<\/span><\/div><\/li>\n<li><div style=\"font: normal normal 1em\/1.2em monospace;margin:0;padding:0;background:none;vertical-align:top\">&nbsp; &nbsp; <span style=\"color: #003399\">System<\/span>.<span style=\"color: #006633\">out<\/span>.<span style=\"color: #006633\">println<\/span><span style=\"color: #009900\">&#040;<\/span>value<span style=\"color: #009900\">&#041;<\/span><span style=\"color: #339933\">;<\/span><\/div><\/li>\n<li><div style=\"font: normal normal 1em\/1.2em monospace;margin:0;padding:0;background:none;vertical-align:top\">&nbsp; <span style=\"color: #009900\">&#125;<\/span><\/div><\/li>\n<li><div style=\"font: normal normal 1em\/1.2em monospace;margin:0;padding:0;background:none;vertical-align:top\"><span style=\"color: #009900\">&#125;<\/span><span style=\"color: #009900\">&#041;<\/span><span style=\"color: #339933\">;<\/span><\/div><\/li>\n<\/ol>\t<\/div>\n\n<\/div>\n\n<p>The collect() function may abort the collecting operation part way through by throwing a CollectionAbortedException.<\/p>\n<p>The disadvantage to this approach is that it is a little less convenient from the perspective of the calling code. The function has a novel pattern, taking ResultHandler&lt;&gt; as a parameter rather than the more familiar Iterable&lt;&gt; as a return value.&nbsp; Logic has to be embedded inside an anonymous or specially constructed ResultHandler&lt;&gt; class.<\/p>\n<p>Unlike the case of returned Iterables, the calling code can not use &#8216;for each&#8217; loops on the result. Nor can it store or pass around the Iterable&lt;&gt; to other systems as a parameter.<\/p>\n<h3>Solution : The yield adapter<\/h3>\n<p>A best-of-both-worlds solution would allow a collecting method based on ResultHandler&lt;&gt; to be automatically adapted at run time to an implementation based on Iterable&lt;&gt;. A data-collecting function implemented in this form (convenient for the collecting code) &#8230;<\/p>\n<div id=\"ig-sh-6\" class=\"syntax_hilite\">\n\n\t\n\t<div class=\"code\">\n\t\t<ol class=\"java\" style=\"font-family:monospace\"><li><div style=\"font: normal normal 1em\/1.2em monospace;margin:0;padding:0;background:none;vertical-align:top\"><span style=\"color: #000000;font-weight: bold\">public<\/span> <span style=\"color: #000066;font-weight: bold\">void<\/span> retrieveAll<span style=\"color: #009900\">&#040;<\/span><span style=\"color: #000066;font-weight: bold\">long<\/span> requesterId, ResultHandler<span style=\"color: #339933\">&lt;<\/span>String<span style=\"color: #339933\">&gt;<\/span> collector<span style=\"color: #009900\">&#041;<\/span><span style=\"color: #339933\">;<\/span><\/div><\/li>\n<\/ol>\t<\/div>\n\n<\/div>\n\n<p>.. would be adapted into this form (convenient for the calling code)..<\/p>\n<div id=\"ig-sh-7\" class=\"syntax_hilite\">\n\n\t\n\t<div class=\"code\">\n\t\t<ol class=\"java\" style=\"font-family:monospace\"><li><div style=\"font: normal normal 1em\/1.2em monospace;margin:0;padding:0;background:none;vertical-align:top\"><span style=\"color: #000000;font-weight: bold\">public<\/span> Iterable<span style=\"color: #339933\">&lt;<\/span>String<span style=\"color: #339933\">&gt;<\/span> retrieveAll<span style=\"color: #009900\">&#040;<\/span><span style=\"color: #000066;font-weight: bold\">long<\/span> requesterId<span style=\"color: #009900\">&#041;<\/span><span style=\"color: #339933\">;<\/span><\/div><\/li>\n<\/ol>\t<\/div>\n\n<\/div>\n\n<p>This would provide the same benefit C# programs get from yield return. Namely, the ability for <i>both<\/i> collecting code and calling code to have their own machine state and callstack control throughout the entire collecting and processing operation. It is not necessary for either side to separate logic into methods inside anonymous classes. Collecting code can use complex recursion algorithms. Calling code can store and defer use of the Iterable&lt;&gt; used to collect results, or pass them as parameters to other functions.<\/p>\n<div>\n<table>\n<tbody>\n<tr>\n<td width=16%>\n        Approach\n      <\/td>\n<td width=16%>\n        Collector controls flow\n      <\/td>\n<td width=16%>\n        Caller controls flow\n      <\/td>\n<td width=16%>\n        Requires list\n      <\/td>\n<td width=16%>\n        Caller can abort collect\n      <\/td>\n<td width=16%>\n        Uses new thread<\/td>\n<\/tr>\n<tr>\n<td width=16%>\n        Collector builds and returns list in form of Iterable&lt;&gt;\n      <\/td>\n<td width=16%>\n        yes\n     <\/td>\n<td width=16%>\n        yes\n      <\/td>\n<td width=16%>\n        yes\n      <\/td>\n<td width=16%>\n        no\n      <\/td>\n<td width=16%>\n        no\n      <\/td>\n<\/tr>\n<tr>\n<td width=16%>\n<p>        Collector returns Iterable&lt;&gt; containing logic\n      <\/td>\n<td width=16%>\n        no\n      <\/td>\n<td width=16%>\n        yes\n      <\/td>\n<td width=16%>\n        no\n      <\/td>\n<td width=16%>\n        yes\n      <\/td>\n<td width=16%>\n        no\n      <\/td>\n<\/tr>\n<tr>\n<td width=16%>\n        Caller passes in ResultHandler&lt;&gt;\n      <\/td>\n<td width=16%>\n        yes\n      <\/td>\n<td width=16%>\n        no\n      <\/td>\n<td width=16%>\n        no\n      <\/td>\n<td width=16%>\n        yes\n      <\/td>\n<td width=16%>\n        no\n      <\/td>\n<\/tr>\n<tr>\n<td width=16%>\n        ResultHandler&lt;&gt; wrapped to Iterable&lt;&gt; with yield adapter.\n      <\/td>\n<td width=16%>\n        yes\n      <\/td>\n<td width=16%>\n        yes\n      <\/td>\n<td width=16%>\n        no\n      <\/td>\n<td width=16%>\n        yes\n      <\/td>\n<td width=16%>\n        yes\n      <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<p><\/br><\/p>\n<h2> How it works <\/h2>\n<p>I am not the first engineer to attempt to bring yield to Java. Aviad&nbsp;Ben&nbsp;Dov&#8217;s article in 2007  <a href=\"http:\/\/chaoticjava.com\/posts\/java-yield-return-code-published\/\">http:\/\/chaoticjava.com\/posts\/java-yield-return-code-published\/<\/a> describes a way to do this using bytecode manipulation and classloader modification. My view is that a solution in Java alone would be more practical as a portable library.<\/p>\n<p>My starting premise was that if calling code and collecting code are both to have their own call stack, this could only be achieved with the use of multiple threads.<\/p>\n<p>In addition, I was aware of a Java collection SynchronousQueue which is designed to allow two threads to pass values between each other, each in turn yielding control to the other.<\/p>\n<p>The Yield Adapter simply makes a new custom Iterable&lt;&gt; which is returned to the calling code. The iterable creates iterators on demand, which also starts a new thread for the collection, and makes a SynchronousQueue for communication between the two. Results are wrapped in message object and, .put() into the queue on the collecting side. The iterator side uses .take(). There is also some extra logic to ensure that incomplete reads do not result in leaked resources.<\/p>\n<p>Please note that although the adapter uses threads, the code does not normally need to be &#8216;thread safe&#8217; in the classic sense. That is because there is never a time that both threads are executing simultaneously. One thread always &#8216;yields&#8217; to the other. The exception to this is if multiple iterators were ever in effect at once.<\/p>\n<h2>Source and demos<\/h2>\n<p>The adapter source can be found <a href=\"http:\/\/svn.jimblackler.com\/jimblackler\/trunk\/IdeaProjects\/YieldAdapter\/src\/net\/jimblackler\/Utils\/ThreadedYieldAdapter.java\">here<\/a> with the interfaces employed all <a href=\"http:\/\/svn.jimblackler.com\/jimblackler\/trunk\/IdeaProjects\/YieldAdapter\/src\/net\/jimblackler\/Utils\">here<\/a>.<\/p>\n<p><a href=\"http:\/\/svn.jimblackler.com\/jimblackler\/trunk\/IdeaProjects\/TestYieldAdapter\/src\/net\/jimblackler\/Utils\/test\/DirectoryScanDemo.java\">A demo here<\/a> shows how the adapter can be used to allow a recursive scan of files and directories, returning the result through an iterator.<\/p>\n<p><a href=\"http:\/\/svn.jimblackler.com\/jimblackler\/trunk\/IdeaProjects\/TestYieldAdapter\/src\/net\/jimblackler\/Utils\/test\/AnagramsDemo.java\">A more complex demo here<\/a> returns all possible ways the letters in a word can be rearranged, without repeats.<\/p>\n<p>The library source can be downloaded with Subversion or your browser <a href=\"http:\/\/svn.jimblackler.com\/jimblackler\/trunk\/IdeaProjects\/YieldAdapter\/\">here<\/a> and the tests and demos are <a href=\"http:\/\/svn.jimblackler.com\/jimblackler\/trunk\/IdeaProjects\/TestYieldAdapter\/\">here<\/a>.<\/p>\n<h2>Feedback<\/h2>\n<p>I hope that this small library will allow people to develop complex algorithms that present results to calling code as iterators.<\/p>\n<p>If you have any comments on the code, please add them as comments on this article.<\/p>\n<h2>Update Feb 2009<\/h1>\n<p>Many thanks to Dominic Lachowicz who brought to my attention the fact that the SynchronousQueue does not in fact stop both threads running at the same time. A revision checked in today uses a Semaphore object to achieve the desired behaviour.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Consider a function that collects and returns a list of results. It might look like this: public ArrayList&lt;String&gt; retrieveAll&#040;long requesterId&#041;; Or even better, this, so it could use anything that implements Iterable&lt;&gt;: public Iterable&lt;String&gt; retrieveAll&#040;long requesterId&#041;; However, storing the results in a container which is then returned may be inefficient for the application. It may [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[],"class_list":["post-61","post","type-post","status-publish","format-standard","hentry","category-general-programming"],"_links":{"self":[{"href":"https:\/\/jimblackler.com\/index.php?rest_route=\/wp\/v2\/posts\/61","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jimblackler.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jimblackler.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jimblackler.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/jimblackler.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=61"}],"version-history":[{"count":0,"href":"https:\/\/jimblackler.com\/index.php?rest_route=\/wp\/v2\/posts\/61\/revisions"}],"wp:attachment":[{"href":"https:\/\/jimblackler.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=61"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jimblackler.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=61"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jimblackler.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=61"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}