USENIX Security '23 Fall Paper #249 Reviews and Comments =========================================================================== Paper #249 Greenhouse: Single-Service Rehosting of Linux-Based Firmware Binaries in User-Space Emulation Review #249A =========================================================================== Paper summary ------------- This paper observes that most vulnerabilities in firmware are unrelated to peripherals. Thus, when applying dynamic analysis like fuzzing, it is unnecessary to always perform the expensive full-system emulation. To this end, this paper presents Greenhouse, a single-service rehosting method for Linux-based firmware in user-space emulation, which is much more efficient than full-system emulation. First, Greenhouse extracts service-related binaries, configurations, and libraries from the firmware. Then, Greenhouse tries to emulate the service binary in the user space and automatically fix the issues that cause emulation failures, named roadblocks in this paper. Specifically, this paper summarizes eight major roadblocks and proposes ten interventions and three binary patching strategies to overcome these roadblocks. In the evaluation, Greenhouse successfully rehosts 2,055 web services in 7,111 firmware. Through case studies, this paper shows that single-service rehosting can deal with firmware that full system emulation-based methods cannot. By replaying exploits and performing fuzzing, the authors show that the fidelity of Greenhouse-rehosted firmware is enough for security analysis tasks like vulnerability risk assessment and vulnerability discovery. Moreover, Greenhouse discovers 24 zero-day vulnerabilities through fuzzing and achieves a 3x fuzzing speedup compared to FirmAFL. Detailed comments for authors ----------------------------- It is a reasonable observation that most vulnerabilities in firmware are unrelated to peripherals. It is also quite realistic that the authors propose user-space single-service rehosting to substitute for the traditional full system emulation-based rehosting. I much like the high-level idea. However, I also have the following concerns about the paper, which prevent me from accepting this paper. First of all, I really appreciate that the authors pay much effort into summarizing the roadblocks and overcoming them case by case. However, I suspect the extensibility of the case-by-case solution. It is unclear how the summarized intervention and patching strategies work on the whole dataset. According to the evaluation, there are still a large number of firmware services that cannot be correctly rehosted. It seems that such a case-by-case fixing approach is still a little far from building a robust user-space single-oriented rehosting system. Besides, I recommend the authors to further discuss the boundary of the current approach, such as which kind of services can/cannot be rehosted, what is the ratio of them in the dataset, and why they can/cannot be rehosted. Furthermore, some details of the approach have not been clearly described. - To check whether a firmware service is rehosted to the “Interact” level, the authors say that the Checker would try common login protocols. What are the “common login protocols”? How to recognize the protocol that a web server uses? How to obtain the necessary information like username and password? - The roadblock R8 is named “Non-critical environment checks”. Why they are “non-critical”? What are the “critical checks”? - The intervention I5, Runtime ArgParser, uses heuristics-based regex pattern matching to find the possible arguments to start the web server. It is unclear how the regex pattern matching extracts the web server arguments. - The authors propose Crash Instruction Patch to eliminate the segmentation faults in the emulation progress. Would this strategy conceal some real vulnerabilities that also cause segmentation faults? The design of the third experiment (Section 7.5) is somewhat confusing. My interpretation of this experiment is to assess the fidelity of the rehosted services. However, I think merely reporting the absolute numbers of successful cases is insufficient to support the conclusion that “Greenhouse-rehosted services are sufficient to use for vulnerability risk assessment”. It is still unclear how is the success rate. Some results in the fourth experiment (Section 7.6) need further explanation. - The fuzzing with Greenhouse discovers 37 unique crashes, of which 24 are zero-day vulnerabilities. What are the remaining 13 cases? Are those crashes caused by rehosting gap introduced by Greenhouse? - Greenhouse’s rehosting bypasses many environment and configuration checks. Does it harm the vulnerability discovery ability? For example, fuzzing with Greenhouse may omit some vulnerabilities that can be found under full-system emulation but cannot be found under user-space emulation, and may raise false positives that will never occur in the real world. Besides, I find another similar work, EQUAFL [A1] that also uses user-space binary rehosting. The idea seems to be the same. From the results, it seems EQUAFL achieves a higher fuzzing speedup ratio than Greenhouse. I recommend the authors compare Greenhouse with this work. There are also some writing issues. - The title of Section 2.2 (Rehosting Fidelity) and Section 2.3 (Rehosting and Execution Fidelity) are quite similar. It would be better to clearly differentiate them. - The “Interact” service numbers in Table 2 and Table 3 are inconsistent. For example, there are 684 services on ASUS firmware rehosted to “Interact”. However, in Table 3, the number becomes 681 (680+1). - The “Total” number in Table 3 is wrong. - The authors discuss the exploit replaying results in both Section 7.5 and Section 7.6.3. These two experiments both aim to measure the fidelity of Greenhouse-rehosted firmware. I suggest the authors combine them together. [A1] Zheng, Yaowen, et al. "Efficient greybox fuzzing of applications in Linux-based IoT devices via enhanced user-mode emulation." Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. 2022. https://github.com/zyw-200/EQUAFL Reasons to accept the paper --------------------------- * Greenhouse successfully rehosts some services that full system emulation-based methods cannot. * The fuzzing efficiency with the proposed rehosting method achieves a significant speedup. Reasons to not accept the paper ------------------------------- * The scope and the scientific contributions of the approach are unclear. * Some important technical details have not been clearly described. * The evaluation part requires further improvement and clarification. * Missing an important related work. Recommended decision -------------------- 4. Reject Writing quality --------------- 3. Adequate Reviewer confidence ------------------- 2. Fairly confident Review #249B =========================================================================== Paper summary ------------- This paper proposes an approach to emulate services of IoT firmware images in user space, instead of running the entire firmware with full-system emulation. To achieve this, the authors utilized several heuristics that resolve emulation failures. Detailed comments for authors ----------------------------- First of all, I agree with the research direction of the paper that focuses on user-space emulation. With user-space emulation, we can save significant resources for security analysis, mostly fuzzing. I appreciate the authors for their efforts for testing a large number of firmware images. While I am positive about the direction of this approach, there are remaining concerns (but not small) to be addressed. # Clear separation of the contributions. First, this paper needs to clearly separate the contributions from the previous studies. I found many components that the authors could (heavily) leverage from Firmadyne and FirmAE for implementing Greenhouse. Furthermore, many parts of the milestones have already been issued and addressed by previous studies. Here, I state some examples (but there are many others): - Containerization is provided by FirmAE. - Firmware unpacking is provided by Firmadyne and FirmAE. - Collecting syscall traces and engines is provided by Firmadyne and FirmAE. - Several “interventions” are provided by Firmadyne and FirmAE (e.g., NVRAM configuration, peripheral accesses, hard-coded network devices, creating dummy networks, using default network setup, and so on) - Status checking routine is provided by FirmAE (although the check routine requires a simple fix) - Selenium-based login function is provided by FirmAE. Please properly cite the contributions in the right place and value previous studies. A summary table that contains the proposed heuristics and comparison with previous works would be appreciated. Additionally, I think Milestone 1 and 4 can be done by simply leveraging FirmAE with a slight modification (e.g., add a check of response data in the status checker.). # Clarification of heuristics for user-space emulation Among the four milestones, I believe that the key contribution of this paper lies in Milestone 2 and 3. Among the proposed heuristics, it is not clear whether a certain heuristic is necessary for 1) switching the full-system emulation to user-space (i.e., qemu-system -> qemu-user), or 2) emulation itself. Categorizing the heuristics would be helpful to understand this. # Lack of details in evaluation I would like to see more analytics and experimental results as follows: - The capability of each heuristics (See FirmAE’s Figure 2 and Table 5). - Need to separate the large dataset vs. better result. How much did you improve in comparison with the result of FirmAE? - Detailed statistics of firmware dataset. (See Table I, II, III, IV in Firmadyne and Table 4 in FirmAE). - Categorizing the emulation results. For example, the authors can present old firmware images and the latest ones separately (See Table 1 in FirmAE). This is important for a large-scale emulation because a single heuristic can boost the emulation rate and can bias the final results within a single vendor and a single model. Thus, adding the number of unique device models is also required. The authors can also test other device categories instead of routers. I am aware that the FirmAE’s dataset includes IP cameras. - The authors only focused on web servers, but, there are many other services in firmware images. Could you test other daemons/services, including CGI binaries? - For vulnerability discovery, the authors only presented the final numbers. Please provide the number of unique vulnerabilities and affected device models as well. (See Table VI in Firmadyne and Table 2, 3 in FirmAE). Please provide - The result in Table 4 is inconsistent with the FirmAE’s paper, particularly for “Path Traversals”. Please double check this. Required changes ---------------- I have listed many in the review. Reasons to accept the paper --------------------------- - Better emulation result? (is it really?) - User space emulation (easier fuzzing) Reasons to not accept the paper ------------------------------- - unclear contribution (did not state exactly what was done before and what they did) - unclear evaluation Recommended decision -------------------- 3. Accept Conditional on Major Revision Writing quality --------------- 2. Well-written Reviewer confidence ------------------- 4. Certain Review #249C =========================================================================== Paper summary ------------- This paper presents Greenhouse, an automated system for large-scale single-service rehosting of Linux-based firmware in user-space. The proposed method tackles the challenges of emulating peripherals for IoT firmware rehosting. The problem with this high rehosting fidelity is that state-of-the-art techniques only allow the analysis of firmware based on small embedded software platforms or baremetal blobs. These current approaches cannot scale to the analysis of more complex Linux-based firmware, the OS used by 43% of IoT devices. The authors argue that we don't need fidelity since only a small percentage of firmware vulnerabilities are hardware related. So it is unnecessary to achieve high-fidelity emulation if the vulnerable system can run without them. The new approach in the paper is user-space automated single-service rehosting for IoT systems with Linux. The paper proposes techniques that automatically find execution barriers during rehosting and use a toolkit of interventions (e.g., eliminate environment checks). Greenhouse complements state-of-the-art by rehosting mostly different firmware. They rehost 2,055 web servers from over 7k firmware images collected. Detailed comments for authors ----------------------------- By limiting the analysis to only Linux-based IoT devices, the problem becomes much simpler as problems such as filesystems and the use of Binwalk is fairly straightforward. The paper avoids the challenges of high-fidelity emulation to focus on a low-hanging fruit problem. Having said that, the method of systematically identifying roadblocks and patching the systems with clearly defined interventions is an interesting contribution. In addition, Greenhouse appears to automatically discover several vulnerabilities with low overhead. So while the problem is easier to solve, it still addresses a problem for the large-scale analysis of Linux-based IoT devices. Reasons to accept the paper --------------------------- - Clearly defined roadblocks and interventions - Rehosts a large quantity of IoT device firmware and finds vulnerabilities in them Reasons to not accept the paper ------------------------------- - The problem and the challenges associated with rehosted are simplified and not as intellectually demanding as some of the previous work. Recommended decision -------------------- 2. Accept on Shepherd Approval Writing quality --------------- 2. Well-written Reviewer confidence ------------------- 2. Fairly confident Review #249D =========================================================================== Paper summary ------------- This paper presents Greenhouse, a rehosting approach for specific services embedded in linux-based firmware. Greenhouse uses user-mode emulation and chroots into the target file system for running the desired service. To circumvent rehosting failures, the authors define heuristics, called interventions, to bypass common roadblocks and deploy an automated patching system as a fall-back. The authors evaluate Greenhouse on the example of web servers and perform both fuzzing of, and replay of known exploits against, successfully rehosted web servers. The results suggest that Greenhouse complements the state of the art and improves fuzzing performance. Detailed comments for authors ----------------------------- Thank you for submitting this paper. I appreciate the commitment to open science, and really enjoyed reading the sections on rehosting fidelity, as well as the automated patching and intervention system. However, I think that the paper has multiple issues which should be addressed before publication. My main problem with the paper in its current form is its positioning. It proposes single-service rehosting as a generic rehosting solution, but then only analyses web interfaces. While I highly appreciate the iterative rehosting process to circumvent roadblocks, some part of Greenhouse seem to be highly geared towards web servers (for instance, the selenium based checking approach or the IPv6 Workaround (I8)). To this end, I think the paper would be stronger if it would demonstrate that the presented rehosting approach can be applied to other targets than web servers. In my opinion, this is especially important, as the paper misses to properly acknowledge the similarities to prior work. In particular, Costin et al. [7] already conducted a large scale study on web interfaces 2016, and, in contrast to what this paper suggests, did not rely on full-system emulation. Instead, they chroot in the target environment with qemu-user, just like Greenhouse. They further define different steps in their analysis, similar to Greenhouse's milestones, and discuss heuristics to improve emulation, similar to interventions. As such, in its current form, Greenhouse seems incremental compared to [7] and I would suggest the authors to properly highlight the differences. Beyond properly distinguishing to [7], the paper also miscategorized prior work, and I would highly encourage the authors to rework Table 1. For instance, Frankenstein [27], and Fuzzware [28] do not rely on full-system emulation (which is suggested by the term "Complete Emulation w/ peripherals") and most systems in this category do *not* require device access. Likewise, "partial emulation" is used in the literature as term to describe hardware-in-the-loop emulation (e.g., in [11, 12, 24]), and it may be good to reframe Greenhouse as "Single Service Emulation" to avoid confusion. Additionally, I would suggest the authors to (a) include [7] in the table, and (b) consider to differentiate Type-I from Type-II/Type-III solutions as the requirements and used techniques are just different. To sum up, I believe that the paper proposes an interesting and promising approach, but requires additional experiments and rewriting to better highlight the differences and improvements to prior work. Minor comments: - I am unsure about the role of FirmAE. Figure 2 suggests that FirmAE interacts with the extraction step, while Section 6.2 suggests that a snapshot from FirmAE is just a possible intervention. I would suggest clarifying. - I would highly recommend the authors to base their experiments on AFL++, rather than legacy AFL, as especially qemu usermode performance has been highly improved. - I would recommend to not only measure the found crashes, but also reached basic blocks over time. Similarly, it may make sense to repeat the experiments multiple times and show statistical significance, as suggested by [R1]. [R1] Klees et al. "Evaluating Fuzz Testing." CCS'18 Required changes ---------------- - Demonstrate that single-service rehosting can scale to other use cases than web servers - Properly discuss prior work, and especially Costin et al. [7] - Fix the representation of prior work in your paper - Consider using AFL++ for your fuzzing campaign Reasons to accept the paper --------------------------- - Nice discussion of different fidelity types - Promising intervention strategies - Improved results compared to the state of the art - Commitment to open source and science Reasons to not accept the paper ------------------------------- - Scaling of approach beyond web servers not demonstrated - Incremental to prior work which is not properly acknowledged - Frequent miscategorization of prior work Recommended decision -------------------- 3. Accept Conditional on Major Revision Questions for authors' response ------------------------------- - Is FirmAE file snapshotting a core part of Greenhouse, or selectively run as intervention? - Which other services could Greenhouse target beyond web servers? Writing quality --------------- 3. Adequate Reviewer confidence ------------------- 3. Highly confident (would try to convince others) Review #249E =========================================================================== Paper summary ------------- The main contribution of this work is a collection and evaluation of techniques that enable rehosting and analysis of independent webserver processes running on Linux router firmware images in emulated Linux userspace processes. Detailed comments for authors ----------------------------- Thanks to the authors for this work addressing a practical problem. I found some interesting insights in this paper, such as that Greenhouse is able to rehost a different set of binaries compared to prior work, and that the interventions are able to practically address patterns causing emulation failure for webservers and enable analysis to find new vulnerabilities. My main concern with this work is the limited novelty, applicability, and overall contribution. I do appreciate the authors' efforts in solving a practical problem. However, the techniques for intervention, which I see as the main contribution of this work, are composed of various basic and well-known techniques, that while effective, are not conceptually novel. In fact, many of the interventions, such as NVRAM handling, seem identical to those in FirmAE, although the authors make no reference. The work is also limited to "Type I" router binaries that run webservers on Linux, thereby limiting its overall applicability (definitely not "Linux-based Firmware Binaries" as the title claims). Overall, while the evaluation does have some interesting insights, the contributions of this paper are currently limited. The main technical area for improvement is the evaluation. The authors need to do a broader evaluation and interpret results more deeply. A study of the impact of interventions on Greenhouse's fidelity is lacking. In Section 7.6.2, the authors say "24 out of 37 are legitimate 0-day vulnerabilities in firmware services". How did the authors confirm that the 24 are are legitimate 0-day vulnerabilities? Are the rest (13) false positives? If so, what were the exact causes of infidelity that led to these false positives? The sample set for Vulnerabiity Risk Assessment in Section 7.5 is unclear (apologies if I missed this). How many firmware binaries did the authors use in this Section's evaluation? Further, the sample set for fuzzing evaluation (10 binaries) is extremely limited and needs to be broader. Overall, I recommend that the authors find ways to improve the paper's contribution. For example, expanding on the class of binaries they support to include Type 2 and Type 3, and better qualification of the limitations of Greenhouse and where it works well. Nits. * Section 7.6.3 - "FirmAFL reported 15 CVEs over nine unique firmware images, three of which we discovered via fuzzing faster than FirmAFL". Should the first word be Greenhouse? Reasons to accept the paper --------------------------- + Solves practical problem + Evaluation found new vulnerabilities Reasons to not accept the paper ------------------------------- - Limited overall novelty, applicability, and contribution - Very shallow evaluation Recommended decision -------------------- 4. Reject Writing quality --------------- 2. Well-written Reviewer confidence ------------------- 2. Fairly confident AuthorFeedback Response by Author [Hui Jun Tay ] (694 words) --------------------------------------------------------------------------- We thank the reviewers for their thorough and insightful feedback and would like to address the following concerns: * __EQUAFL__ (A) We will extend our fuzzing evaluation to include EQUAFL and their dataset of 70 images. EQUAFL is not a rehosting solution but rather a fuzzing one. Greenhouse differs from EQUAFL as follows: 1) The EQUAFL authors only included a small subset of samples collected that Firmadyne could already emulate (EQUAFL Section 6.1). EQUAFL then transforms these rehosted firmware images into a user-space emulation. Their rehosting capabilities are thus capped by the full-system emulation technique. Meanwhile, Greenhouse can rehost in user-space firmware images that FirmAE, an improvement on Firmadyne, cannot. 2) Greenhouse uses patching and other interventions to rehost a wider variety of targets while EQUAFL does not. We found ~55/70 EQUAFL samples inside Greenhouse’s dataset, of which we rehost 37. 3) EQUAFL is not a generic solution: their implementation contains hardcoded, manually-written hooks to fix rehosted functionality for at least 9 of their 66 fuzzable samples. Greenhouse’s user-space fuzzing does not require any target-specific logic. * __Attribution and categorization of related work__ (A,B,C,D,E) In our revision, we will fix miscategorizations, improve the differentiation between the contributions of our paper from contributions in related work, and credit the authors accordingly. We will add a table to show how our interventions and heuristics compare with related work. Section 5, “Rehosting Milestones,” discusses a metric for measuring the fidelity of a rehosted emulation. It is not a summary of contributions by Greenhouse. * __Scientific contributions of Greenhouse__ (A,B,C,D,E) Greenhouse presents two unique scientific contributions: 1) our use of binary patching to rehost firmware services in user-space; 2) adapting interventions to user-space emulation and applying them iteratively to minimize fidelity loss. Following Costin’s approach, we performed a preliminary baseline evaluation with no interventions, finding ~715/7,111 samples with network activity of which only 175 returned HTTP 200, suggesting the impact of our iterative interventions is significant. * __Further breakdown of experiments__ (B) Per space limit, we will provide tables in the appendix showing more detailed statistics of the dataset, including a breakdown of rehosting results by number of unique devices, and a breakdown of the contribution of each intervention during rehosting. * __Evaluation is entirely webservers__ (B,D) We focused on webservers to enable an apples-to-apples comparison with related work like FirmAE, Firmadyne, and FirmAFL that also targeted webservers and used network activity as a success metric in their evaluation. However, this is a surface implementation detail: In our revision, we will evaluate Greenhouse’s ability to rehost other border-binary services in our dataset. * __Improvements to fuzzing evaluation__ (D,E) We will implement the suggestions to measure basic-block coverage over time and repeat our fuzzing runs for statistical significance. We used AFL++ in our fuzzing evaluation, mistakenly labeled as AFL in the submission. * __13 crashes did not become unique vulnerabilities__ (A,E) The 13 crashes are not false positives created by Greenhouse, but crashes that trigger the same vulnerability (root cause) as another crash despite significantly different coverage. We will better explain these crashes in our revision. * __False-positives__ (A,E) While we did not observe any false-positive crashes that Greenhouse caused during our evaluation, we will add a section discussing potential false positives caused by interventions (like binary patching) that may incur significant fidelity loss. * __Interaction with FirmAE__ (D) Using a filesystem snapshot from a FirmAE emulation is an _optional_ intervention that Greenhouse applies at the extraction stage after binwalk. Greenhouse does not require a successful FirmAE emulation to rehost in user-space. In our revision, we will report the percentage of firmware samples that required FirmAE-based intervention. * __Readability__ (A,C,D) We will fix the inconsistencies and labeling mistakes, update confusing terminology, and reduce section overlap. --- We propose a list of actionables for our revision: 1. Compare against EQUAFL, and evaluate fuzzing performance of Greenhouse vs EQUAFL on our dataset using suggested fuzzing improvements. 2. Rehosting non-webserver border-binary services. 3. Correct mistakes in attribution and categorization of related work. 4. Clarify differences between Greenhouse and related work. 5. Add tables showing more detailed statistics (unique devices and individual interventions). 6. Discuss potential false positives caused by patching. 7. Clarify why not all unique crashes become unique vulnerabilities. 8. Improve writing. Comment @A2 --------------------------------------------------------------------------- Revision list: - Clarify whether a specific heuristic is necessary for 1) switching the full-system emulation to user space (i.e., qemu-system -> qemu-user), or 2) emulation itself. - The authors need to show that Greenhouse can be applied to targets other than web servers. The revision should not focus only on the web server. - Compare against EQUAFL, and evaluate the fuzzing performance of Greenhouse vs. EQUAFL using suggested fuzzing improvements. - For fuzzing experiments, also run multiple repetitions of 24 hours for the targets in Table 6. Report on reached coverage over time, especially for the comparison versus FirmAFL/EQUAFL. - Correct mistakes in attribution and categorization of related work. - Clarify differences between Greenhouse and related work. - Add tables showing more detailed statistics (unique devices and individual interventions). - Discuss potential false positives caused by patching. - Clarify why not all unique crashes become unique vulnerabilities. - Improve writing. Comment @A3 by Author [Hui Jun Tay ] --------------------------------------------------------------------------- Thank you for the decision on Major Revision and the helpful feedback provided! We will present a detailed revision plan that fulfills the criteria outlined in the revision list. The detailed plan will be uploaded here as soon as it is ready. Comment @A4 by Author [Hui Jun Tay ] --------------------------------------------------------------------------- **Clarify whether a specific heuristic is necessary for 1) switching the full-system emulation to user space (i.e., qemu-system -> qemu-user), or 2) emulation itself.** We are unclear on what "switching the full-system emulation to user space" means and would appreciate a clarification from the reviewers regarding this requirement. Our understanding is that the reviewers expect us to detail (1) what interventions (heuristics) are needed to address the differences between rehosting an image inside a full-system emulation (like in FirmAE) vs a user-space one (as in Greenhouse), and (2) what interventions are simply part of emulation outside of the device (e.g. roadblocks caused by NVRAM accesses). Thank you! Comment @A5 by Shepherd --------------------------------------------------------------------------- Yes, please consider the first bullet point of the revision requirements to be the following: Detail (1) what interventions (heuristics) are needed to address the differences between rehosting an image inside a full-system emulation (like in FirmAE) vs a user-space one (as in Greenhouse), and (2) what interventions are simply part of emulation outside of the device (e.g. roadblocks caused by NVRAM accesses). Comment @A6 by Author [Hui Jun Tay ] --------------------------------------------------------------------------- Thank you for the clarification! We are preparing the detailed revision plan and will post it soon. Comment @A7 by Author [Hui Jun Tay ] --------------------------------------------------------------------------- Thank you for the feedback, below is our detailed revision plan: **R1) Clarify whether a specific heuristic is necessary for 1) switching the full-system emulation to user space (i.e., qemu-system -> qemu-user), or 2) emulation itself.** _Writing Changes:_ - we will detail which interventions (under Section 6 Roadblocks and Interventions) are responsible for handling roadblocks that arise specifically due to rehosting in user-space instead of a full-system emulation, and which interventions are part of enabling emulation outside of the original device. **R2) The authors need to show that Greenhouse can be applied to targets other than web servers. The revision should not focus only on the web server.** _New Experiments:_ - to address this, we will target other border-binary services found inside our dataset and present the rehosted numbers. - we will also add at least one case study of a border-binary service to show Greenhouse’s generalizability beyond web servers (under Section 7 Evaluation). - we will use the same approach in SaTC to identify border-binary services in our dataset. The distribution of these binaries will be reported accordingly in the Appendix. - we will demonstrate the extensibility of our design by adding simple Checker plugins for selected border-binary services to test for interaction, which will be discussed in our case study. **R3) Compare against EQUAFL and evaluate the fuzzing performance of Greenhouse vs. EQUAFL using suggested fuzzing improvements.** _Writing Changes:_ - we will add a section under Section 2 (Background and Motivation) to discuss EQUAFL and how Greenhouse differs from EQUAFL. - we will overhaul section 7.6 (Fuzzing Rehosted Services) to include comparisons against EQUAFL. _New Experiments:_ - we will conduct all evaluations against publicly available EQUAFL artifacts. - EQUAFL's publicly available dataset seems to contain a few images that are not in Greenhouse's dataset (namely European versions) which we will add to our run. - in cases where information on the exact version cannot be found, we will use the latest version. - we will evaluate fuzzing performance against the nine samples listed in Table 4 of the EQUAFL paper. - we will evaluate EQUAFL on our full dataset and compare its rehosting performance against Greenhouse. - we will evaluate Greenhouse on the 70 samples in the EQUAFL dataset and compare the rehosting performance against EQUAFL. - we will evaluate the fuzzability of Greenhouse's samples, as per the metric in EQUAFL's Table 3 (SUCC, ERR, HAN, CRA), on all rehosted images Greenhouse can produce, with a focus on the 70 samples that overlap with Greenhouse's rehosted images. **R4) For fuzzing experiments, also run multiple repetitions of 24 hours for the targets in Table 6. Report on reached coverage over time, especially for the comparison versus FirmAFL/EQUAFL.** _New Experiments:_ - we will rerun our fuzzing evaluations for EQUAFL and FirmAFL as discussed under R3) with these improvements. **R5) Correct mistakes in attribution and categorization of related work.** **R6) Clarify differences between Greenhouse and related work.** _Writing Changes:_ - fix Table 1 (categorization of previous work) to accurately categorize previous tools. - rewrite Section 2 (Background and Motivation) to address differences between Greenhouse, EQUAFL, and Costin et al. and better position our work. - include discussion of Costin et al. and EQUAFL in Section 9 (Related Work) - check routersploit results for FirmAE under Table 4 (Breakdown of N-day replay results) - rewrite Section 6 (Roadblocks and Interventions) to better attribute interventions to related work and highlight differences in how Greenhouse applies these interventions to a user-space environment (also done as part of addressing R1). - attribute clearly in Section 6 (Roadblocks and Interventions) what components of Greenhouse are extensions of previous work (e.g. Selenium-based login function is extended from FirmAE) and the significance of these extensions. - rewrite Section 4 (Greenhouse Overview) and 5 (Rehosting Milestones) to address confusing wording, compare Greenhouse’s rehosting stages to related work and make clear differences between metrics established by Greenhouse and metrics in related work. - adjust phrasing and writing for Section 5 (Rehosting Milestones) to make clear these are not contribution milestones but a metric for measuring rehosting. - clarify differences in requirements under Section 2 (Background and Motivation) for Type I vs Type II/III work to make clear limitations. **R7) Add tables showing more detailed statistics (unique devices and individual interventions).** _New Experiments:_ - we will rerun Greenhouse multiple times on the full sample set with one of the following interventions disabled during each run: - a) no File Setup (I1) - b) no File Sanitization (I2) - c) no File-System Snapshot (I3) - demonstrates Greenhouse capabilities without FirmAE Snapshot - d) no NVRAM-faker support (I4) - e) no ArgParser (I5) and no File-System Snapshot (I3) - demonstrates impact of ArgParser that is used when File-System Snapshot fails - f) no Dummy Network Devices (I6) - g) no Background Script Plugins (I7) - h) no IPV6 workarounds (I8) - i) no Automated Patching - sysinfo (I9) and Logging Behavior (I10) changes are necessary interventions that enable feedback mechanisms in Greenhouse, so we cannot disable them. _Writing Changes:_ - filter and sort data by ”latest” firmware versions (smaller set) for both 1124 FirmAE samples and ~7k Greenhouse samples. - results will be added to tables in the Appendix and discussed in Section 7 (Evaluation). **R8) Discuss potential false positives caused by patching.** _Writing Changes:_ - add subsection under Section 8.2 (Limitations) discussing possible false positives that patching might cause. - because we will perform a larger-scale fuzzing campaign on the EQUAFL dataset, we will also manually investigate as many crashes as possible and analyze any false positive crashes found during fuzzing. **R9) Clarify why not all unique crashes become unique vulnerabilities.** _Writing Changes:_ - we will add a subsection under Section 7.6 (Fuzzing Rehosted Services) discussing Table 6 (vulnerabilities found through fuzzing) after updating with EQUAFL values, explaining how filtering crashes works. - we will include a table in the Appendix detailing statistics for each crash -> vulnerability (what they are, how they are filtered). **R10) Improve writing.** _Writing Changes:_ - differentiate Section 2.2 (Rehosting Fidelity) and 2.3 (Rehosting and Execution Fidelity). - fix inconsistent numbers in Table 2 and Table 3 (Rehosting Results). - adjust writing to clarify differences between Sections 7.5 (Vulnerability Risk Assessment) and 7.6.3 (Replaying Exploits). - reword the start of Section 7.6.3 (Replaying Exploits) to clarify how we are comparing against FirmAFL's CVE discovery. - other phrasing improvements for clarity (e.g. rephrase "partial emulation" as related papers have used similar terms to refer to different things). Comment @A8 by Author [Hui Jun Tay ] --------------------------------------------------------------------------- Please let us know if you have any suggestions on the detailed revision plan above. Comment @A9 by Shepherd --------------------------------------------------------------------------- The detailed plan looks good. I encourage you to submit a preliminary version of the document well before May 25 (you don't need to have all promised experiments done in this first draft) so we can check the writing changes first and provide early feedback on experiments. Comment @A10 by Author [Hui Jun Tay ] --------------------------------------------------------------------------- Thank you for the advice. We will be sure to have a preliminary draft up before then. Comment @A11 by Author [Hui Jun Tay ] --------------------------------------------------------------------------- Update: We've been having some trouble reproducing EQUAFL's results due to issues with their public data set. At the moment, public information about their dataset (https://sites.google.com/view/equafl/dataset) is incomplete and does not provide enough details (e.g., hashes) for us to reliably rebuild their original data set. Based on the information available, we manually curated 54 out of 70 images mentioned in their paper. We have been contacting the authors since January, but have not been able to get more information about the data set. We will continue contacting the authors to request the complete data set. As a fallback, we will build an approximation of their data set based on the incomplete public information in the event that we are unable to get timely help from the original authors. Please let us know if you have any feedback on this plan. Comment @A12 by Shepherd --------------------------------------------------------------------------- We thank the authors for their update. The authors' plan of continuing a best-effort comparison that outlines the limitations of reproducibility from previous work sounds good. Comment @A13 by Shepherd --------------------------------------------------------------------------- I wanted to follow up on our last exchange with the authors. Do you have a tentative timeline for when you plan to submit your draft revision? We would like to have enough time to go over it and give feedback in case there is a need for a second round of revisions. Comment @A14 by Author [Hui Jun Tay ] --------------------------------------------------------------------------- We are currently targeting 12th May for the preliminary submission of our revised draft. The tentative timeline is as follows: - 14th Apr: Completed relabeling of ~7k samples in dataset to discern latest set (R7). As not all crawled images came with metadata, we had to manually determine which images were the latest firmware for a given device. In some cases, the firmware we have is not the latest, and had to be added to the dataset. - 21st Apr: Rehosting statistics for Greenhouse rehosting on new dataset including the latest firmware for each device - 5th May: Address writing issues (R1, R5, R6, R10) - 1st May: Rehosting statistics for Border Binary Experiments to show Greenhouse applies to other targets (R2) - 11th May: Statistics for comparison of fuzzing and rehosting performance against EQUAFL (R3, R4). Discuss false positives caused by patch (R8) and the statistics for found vulnerabilities (R9) - 12th May: Preliminary Draft for review - 22nd May: additional experimental data with various interventions disabled (R7) - 30th May: cutoff for final submission with all experimental data Comment @A15 by Author [Hui Jun Tay ] --------------------------------------------------------------------------- We would like to inform the reviewers of our current progress. We have completed 5/10 major revision requirements outlined in our detailed plan (**R1, R2, R4, R5, R6**) - we have rewritten Section 6 to clarify which roadblocks/interventions are related to user-space emulation, and which apply to both user-mode and full-system emulation, citing previous works accordingly. (**R1**) - we have run Greenhouse on UPNP and DNS web services in our dataset and obtained the rehosting results to show we are not limited to HTTP web servers. (**R2**) - we have recovered as much of EQUAFLs' dataset as possible and completed the performance evaluation of the EQUAFL samples in Table 4 of their paper for both Greenhouse and EQUAFL for multiple repetitions of 24 hours. (**R4**) - we have largely rewritten sections 2,4,5 and 9 based on feedback to better attribute previous works (Costin and EQUAFL especially) and position Greenhouse more clearly. We are working on fixing Table 1 (**R5 ,R6**) We have completed parts of **R3, R7, R10** - we have made the writing changes to sections 2 and 9 to include EQUAFL (partial **R3**) - we have run EQUAFL on the full Greenhouse dataset and obtained its rehosting results (partial **R3**) - we have determined the number of unique devices in our dataset and added the latest version of each (partial **R7**) - we have rerun Greenhouse, FirmAE and EQUAFL on this new dataset and obtained rehosting results for the latest version of each device (partial **R7**) - we have made a number of improvements to phrasing and ambiguity (partial **R10**) Currently, we are waiting for results from the fuzzing experiment for the EQUAFL samples rehosted by Greenhouse - without some preliminary fuzzing results on our new dataset we are unable to complete (**R3, R8 or R9**) - these results will also significantly shape how we rewrite the Section 7 Evaluation, which affects **R10** - as each removed intervention needs to be run on our entire dataset for 48 hours, **R7** will take time to complete We think a more reasonable draft with most if not all the writing changes will be ready by **Thursday, 18th May.** We will try to submit this draft by 18th May, even if not all the numbers for our experiments (**R7** in particular) are complete Comment @A16 by Shepherd --------------------------------------------------------------------------- Thank you for the update. I see that you are a week behind in the revision timeline; however, I understand that the revision list was extensive, and I can understand this delay in finalizing the new experiments. I (the shepherd) will be out of town May 17-22, so I won't be able to give feedback during that period. But we look forward to a prompt submission of the draft so we have enough time for the revision round. Comment @A17 by Author [Hui Jun Tay ] --------------------------------------------------------------------------- Thank you for your patience! Since our last update, we have fully completed **7/10** of the major revision requirements (**R1, R2, R4, R5, R6, R8, R10**) and have largely completed **3/10** of the remaining requirements (**R3, R7, R9**). We are currently still conducting the ablation study (**R7**, running Greenhouse with interaction disabled) and large-scale fuzzing coupled with manual reverse-engineering of any obtained crashes for real-world vulnerabilities (**R3, R9**). As such, Section 7 Evaluation and various tables in the Appendix will be updated. In particular, Table 6 and Table 7 will be updated, and the current values are the most recent, _conservative_ experiment results (i.e., the final numbers will likely be better). We also recognize that some of the numbers for FirmAE and EQUAFL in our experiments do not match their original reported numbers. After some thorough testing, we believe this is caused by some inherent flakiness in the implementation of QEMU-system, likely due to the network components. Constantly rerunning the emulation seems to eventually lead to a success for failed rehosting targets. We will continue to look into the matter. Below is a summary of the changes made and the revision goal they correspond to: **R1) Clarify whether a specific heuristic is necessary for 1) switching the full-system emulation to user space (i.e., qemu-system -> qemu-user), or 2) emulation itself.** We added two paragraphs under Section 6 that specify which roadblocks and interventions were implemented by previous work. We also added a short sentence or two under each roadblock and intervention discussing any additional complications due to user-space emulation **R2) The authors need to show that Greenhouse can be applied to targets other than web servers. The revision should not focus only on the web server.** We have added UPNP and DNS web services to our evaluation. Table 5 in Section 7 details the rehosting numbers for these services. Note that not all firmware samples contain identifiable border binaries that correspond to these services - we used a combination of SaTC and manual curation to create a whitelist of binaries for each service. We listed the % of discovered binaries that we were able to rehost and compare against our success rate for HTTP web services. **R3) Compare against EQUAFL and evaluate the fuzzing performance of Greenhouse vs. EQUAFL using suggested fuzzing improvements.** **R4) For fuzzing experiments, also run multiple repetitions of 24 hours for the targets in Table 6. Report on reached coverage over time, especially for the comparison versus FirmAFL/EQUAFL.** We have added EQUAFL to Table 1 and discussed it under Section 2.3 Existing Approaches and Section 9 Related Work. We have replaced our evaluation against FirmAFL with an evaluation against EQUAFL and overhauled Section 7.6. We added EQUAFL samples to our existing dataset, rehosted them with Greenhouse and fuzzed them. We compared Greenhouse against EQUAFL in 3 ways: We attempted large-scale fuzzing of all Greenhouse rehosted samples and counted the number of targets that we were able to run the web-server for via our simple fork-server fuzzer. Table 7 shows the breakdown under EQUAFL’s fuzzability metric (SUCC, ERR, HAN, CRA), though the numbers are not finalized. We did a performance evaluation of Greenhouse versus EQUAFL for the 6 samples from EQUAFL’s Table 4 and 2 samples from FirmAFL that were part of our previous evaluation. We ran ten repetitions for 24 hours each, measuring the execs per second each hour and taking the resulting average. We also measure the total corpus count of the fuzzing as a function of coverage. This comparison can be found in Figure 3. We attempted to use EQUAFL to rehost our dataset of 7,143 samples. EQUAFL was able to emulate 2,414 images but only rehost 568 to network connectivity, of which 440 did not return any obvious HTTP errors. A more detailed breakdown of our comparison of Greenhouse vs EQUAFL can be found in Section 7.6 and Appendix. **R5) Correct mistakes in attribution and categorization of related work.** We have rewritten Sections 2.2 and 2.3 to discuss EQUAFL and Costin et al. We include specific mention of Costin as another large-scale rehosting study and discuss how it surveys different emulation approaches, including user-space emulation. We note that Costin dismisses user-space emulation as too unstable, and the rest of their study is conducted on a chroot inside a generic kernel emulator, which we consider a subset of full-system emulation. We have also revamped Table 1 based on feedback to better group previous techniques, including the firmware type and a more nuanced division in the fidelity level and primary emulation technique used. **R6) Clarify differences between Greenhouse and related work.** We have added a paragraph to Section 4, Greenhouse overview, that clarifies similarities between Greenhouse and related work, and how we differ from there. We trimmed the subsequent sections to focus mostly on the unique aspects of Greenhouse’s implementation. We renamed Section 5 Milestones to Rehosting Metrics and mention the metrics used by previous work and how Greenhouse’s differ. As per **R1)** we have added text to Section 6 attributing which interventions were based on prior research and what changes Greenhouse made to adapt them to user-space rehosting. **R7) Add tables showing more detailed statistics (unique devices and individual interventions).** We manually curated our dataset to sort by unique device and the latest firmware for a total of 1,768 devices. Greenhouse rehosts 532 of these. A detailed breakdown is in Table 3. Due to the amount of computation resources this requires, we do not have the final results for the various runs with different interventions disabled. We will have a table with all these numbers in the Appendix by 6th June. **R8) Discuss potential false positives caused by patching.** We added a subsection under Section 8.2 discussing possible false positives caused by patching. So far, we have found 2 false positives that could not be replicated manually, neither of which were samples that had been patched. We will continue to investigate the crashes from our large-scale fuzzing run but believe that the manual effort required to process thousands of crashes is not possible within the time-frame of this paper. **R9) Clarify why not all unique crashes become unique vulnerabilities.** We discuss how some crashes found through fuzzing may not become vulnerabilities in Section 8. Once we have finalized our fuzzing results and the vulnerabilities found, we will add a table showing a detailed breakdown of unique crashes to vulnerabilities in the appendix for the subset of rehosted and fuzzed samples we selected for the study. **R10) Improve writing.** We renamed section 2.3 to “Existing Approaches,” fixed inconsistent numbers in the evaluation tables, and renamed and adjusted the writing for section 7.6.3 to be focused on the Large-scale Fuzzing results. We no longer compare against FirmAFL’s CVE discovery process as the nature of the evaluation has changed when we switched to comparing against EQUAFL. We also made various phrasing improvements and adjustments to terminology, such as changing “partial emulation” to “limited emulation” as per feedback. Comment @A18 by Shepherd --------------------------------------------------------------------------- Dear Authors, we reviewed your major revision and are satisfied with the changes. So we are recommending an acceptance to the PC chairs. Congratulations! Comment @A19 by Author [Hui Jun Tay ] --------------------------------------------------------------------------- Thank you for the recommendation! We thank the reviewers and shepherd for their guidance and look forward to presenting at USENIX 2023. Comment @A20 --------------------------------------------------------------------------- Dear Authors, this is one of the co-chairs. Congratulations on your paper! In your decision email from January 27, you will find instructions for preparing a camera-ready version of your paper, including a link here with additional instructions: https://www.usenix.org/conference/usenixsecurity23/instructions-presenters Given the timing of your approval, the prepublication of your paper will occur with papers accepted during the Winter cycle of USENIX Security 2023, and the camera-ready version of your paper is due June 13, 2023. The submission site is not accepting final versions of papers at the moment, but we will reenable this feature approximately ten days before that June 13, 2023 deadline. If you have any questions, feel free to contact us (sec23chairs@usenix.org). Best, Carmela Troncoso