Rewrite HTTP Headers (JS)

tl;dr: All development on my Firefox add-ons has been halted.

Airing of Grievances against Mozilla:

Firefox has announced that their roadmap for the near-future is to completely deprecate the existing add-on API. Doing so will effectively make ALL current add-ons inoperable. The new API is a stripped-down version of Google Chrome's extension API; Mozilla is literally linking its own documentation to a cherry-picked selection of pages on developer.chrome.com.

Prior to that, Firefox had announced that all add-ons must be digitally signed by Mozilla. This particular policy makes Firefox feel more like Apple's App Store than FOSS. Even Android allows side-loading of .apk files, either developed locally or found in the wild; and yet Firefox is building a "walled garden". To complicate this farther, the review process for an add-on to be hosted on AMO can take upwards of a year, and is a very disorganized process. Personally, I've had very bad experiences with abrupt and unfair rejections; my more complicated add-ons have only ever been hosted on github, and they can no-longer be installed by recent releases of Firefox.

I'm currently focusing my time on Android development. If Firefox reverses some of these policies and begins behaving in a way that's more in keeping with its own mission and manifesto, then I would gladly continue to contribute my time and effort to its ecosystem. But until that happens, I'm done with Firefox.. both as a user, and as a developer.



------------------------------------------------------------

Summary

Swiss Army knife for anyone interested in writing rules to intercept and conditionally modify HTTP traffic
a distinct set of rules are written for each:
- (outbound) requests
- (inbound) responses
rule sets are described using a Javascript format
rule sets are stored in external files, and specified by path
paths may be specified relative to "special directories" to enable portability (see: documentation)
external rules files can (optionally) be watched for updates
rules can:
- add/edit/remove HTTP headers
- cancel the request
- redirect the request to a different URL
- save a record of the request
saved requests can:
- be replayed using (supported) external download tools.
  replayed requests include all the HTTP headers and POST data in the original/saved request.

Video Tutorial

2+ hour long screencast that is a detailed walk-through of installation, configuration and usage

Features

regex patterns are used to match rules against the requested URL
rules are applied incrementally, until either:
- all rules have been processed
- a rule is marked as being final
rules are declared within a well-defined data structure.
this data isn't JSON; it is evaluated as javascript.
as such:
- all javascript data types are supported,
  including those that aren't representable using JSON.
  for example:
  - comments
  - regex patterns
    - literal (ie: perl) notation: //
    - object constructor: new RegExp('')
  - functions
- the declaration for the data structure can contain inline code,
  which is interpolated only once during evaluation,
  immediately after the data file is initially read from disk.
  for example:
  - storing the output of a helper function as (part of) a static value
  - calling an "immediately-invoked function expression" (aka: "self-executing anonymous function"), and storing its output as (part of) a static value
where a function is present, it will be called each time the rule is evaluated.
- rules are evaluated for every request and/or response.
- when functions are called, there will be contextual variables as well as helper functions in scope.
- the contextual variables will allow the function to return a value that is dependent upon the state of the request/response.
- the helper functions provide a library to perform tasks that:
  - are commonly used to generate HTTP header values
  - provide enhanced capabilities, unrelated to modifying HTTP header values
where inline javascript code is present, the javascript will only be evaluated once.
- this occurs when the rules are read from an external file and evaluated into a javascript array of rule objects.
- when this evaluation occurs, there is no contextual request or response.. so there are no contextual variables in scope.
- however, the same helper functions that are always available to functions (that are defined within the rules data set) will also be available at the time that the rules data set is initialized/evaluated.

Contextual Variables (in scope when functions are called)

both requests and responses
- request.window_location = {}
  keys:
  - href: [string] full URI
  - protocol: [string]
    for example:
    - http:
    - https:
    - file:
  - username: [string]
  - password: [string]
  - host: [string]
  - port: [string]
  - path: [string]
  - query: [string]
  - hash: [string]
  - file_ext: [string]
- request.uri = {}
  
  same keys as: request.window_location
- request.original_uri = {}
  
  same keys as: request.window_location
- request.referrer = {}
  
  same keys as: request.window_location
- request.method [string]
- request.headers = {}
- request.headers.unmodified = {}
  hash of all HTTP headers in original/unmodified request
- request.headers.updated = {}
  hash of all HTTP headers that the rules array data set (for requests) has incrementally modified at the current point of rules processing.
  - this hash is empty before any rules are processed.
  - as rules are processed in sequential order, any rule that matches the requested URL may specify updated HTTP headers which will be applied to this hash object.
  - when rule processing is completed, the final state of this hash will be applied to the HTTP request.
    - a string value will set/update the HTTP header corresponding to its hash key.
    - a boolean false value will be ignored.
    - a null value will remove the HTTP header corresponding to its hash key (if originally present).
request only
response only
- response.headers = {}
- response.headers.unmodified = {}
  hash of all HTTP headers in original/unmodified response
- response.headers.updated = {}
  hash of all HTTP headers that the rules array data set (for responses) has incrementally modified at the current point of rules processing.
  
  see additional notes under: request.headers.updated
- response.status_code [integer]
- response.charset [string]
- response.content_length [integer]
- response.content_type [string]

Helper Functions (in scope when functions are called)

always available
- atob(string_base64_encoded)
  decodes a string of data which has been encoded using base-64 encoding.
- base64_decode(string_base64_encoded)
  
  alias for: atob
- btoa(string_value)
  creates a base-64 encoded ASCII string from a "string" of binary data.
- base64_encode(string_value)
  
  alias for: btoa
- md2(string_value)
  returns the result of hashing the input string using the md2 crypto hash function
- md5(string_value)
  returns the result of hashing the input string using the md5 crypto hash function
- sha1(string_value)
  returns the result of hashing the input string using the sha1 crypto hash function
- sha256(string_value)
  returns the result of hashing the input string using the sha256 crypto hash function
- sha384(string_value)
  returns the result of hashing the input string using the sha384 crypto hash function
- sha512(string_value)
  returns the result of hashing the input string using the sha512 crypto hash function
both requests and responses
- save()
  prepends a record of the current request to the Output File.
  this record will be available for replay via the view/replay saved requests dialog window.
request only
- redirectTo(string_URI)
  
  hint: window.location = string_URI
  
  for an example, check out the recipe: redirect search engine queries from Yahoo to Google
- cancel()
  completely cancels the request
  
  for an example, check out the recipe: light weight ad-blocker
response only

Data Structure

the same data structure (schema) applies to both requests and responses
request and response data are defined separately
the data structure is an array of objects, where each object represents a rule.
each rule can have the following attributes:
- url (required, RegExp)
- headers (required, object or function):
  - key = name of HTTP header
  - value = determines what action to take
    - [string]:
      new value of HTTP header
    - [boolean, value === false]:
      ignore all previous rules for this HTTP header,
      and leave the original value unmodified
    - [object, value === null]:
      remove HTTP header from request/response
- stop (optional, boolean or function):
  when:
  - the url pattern of this rule matches the requested URL, and
  - processing of this rule is complete
  then:
  - [false, default]: process next rule (in array)
  - [true]: do not process any additional rules
while each request/response is processed by its corresponding rules data set,
when a url pattern match occurs for a rule,
when either the headers or stop attribute of the matching rule object is declared as a javascript function:
- contextual variables will be constructed and made availabe to the function that will convey information about the particular request/response being processed.
- the function is expected to return the proper data type; otherwise, its output will be ignored.
while rules are being processed, an internal list of updates is being created and incrementally updated.
when the processing of rules is complete, this internal list of updates are applied to the request/response.

Simple Examples

sample request rule(s):


[
    /* ****************************************************
     * all requests: add 3 custom headers
     * ****************************************************
     */
    {
        "url" : /^.*$/,
        "headers" : {
            "X-Custom-Sample-Header-01" : "Foo",
            "X-Custom-Sample-Header-02" : "Bar",
            "X-Custom-Sample-Header-03" : "Baz"
        }
    },
    /* ****************************************************
     * secure requests: cancel the 3 custom headers, and stop processing rules
     * ****************************************************
     */
    {
        "url" : /^https/i,
        "headers" : {
            "X-Custom-Sample-Header-01" : false,
            "X-Custom-Sample-Header-02" : false,
            "X-Custom-Sample-Header-03" : false
        },
        "stop": true
    },
    /* ****************************************************
     * all requests: update 1st custom header, cancel 3rd custom header
     * ****************************************************
     */
    {
        "url" : /^.*$/,
        "headers" : {
            "X-Custom-Sample-Header-01" : "Hello",
            "X-Custom-Sample-Header-03" : false
        }
    }
    /* ****************************************************
     * assertion #1: non-secure URL request
     * expected result:
     *     X-Custom-Sample-Header-01: Hello
     *     X-Custom-Sample-Header-02: Bar
     * ****************************************************
     */
]

sample response rule(s):


[
    /* ****************************************************
     * purpose: map an applicable 'content-type' to a finite set of resources
     *          as identified by file extension, when loaded from local hard disk.
     * ****************************************************
     */
    {
        "url" : new RegExp('^file://', 'i'),
        "headers" : function(){
            var $headers = {};
            switch( request.uri.file_ext.toLowerCase() ){
                case 'txt':
                    $headers['content-type'] = 'text/plain';
                    break;
                case 'css':
                    $headers['content-type'] = 'text/css';
                    break;
                case 'js':
                    $headers['content-type'] = 'application/javascript';
                    break;
                case 'json':
                    $headers['content-type'] = 'application/json';
                    break;
            }
            if ( $headers['content-type'] ){
                response.content_type = $headers['content-type'];
            }
            return $headers;
        },
        "stop": true
    }
]

More Complicated Examples

a collection of various interesting rules and useful examples has been dubbed the recipe book
it can be found in its own branch of this repo, named: js/data/recipe-book
users are encouraged to contribute (via push request) additional recipe examples

User Preferences

input: rules data
- HTTP Requests (outbound):
  - Enabled
    
    default: on
    
    on/off toggle
    - on:
      intercept HTTP Requests and apply its corresponding set of rules
    - off:
      disable this feature entirely
  - Path to Rules File
    
    default: ''
    
    refer to Comments / Implementation Notes for advanced usage
  - Watch Interval (ms, 0 to disable)
    
    default: 0 (off)
    
    useful while writing/testing new rules.
    this feature will watch the rules file for changes, and reload its contents as needed.
- HTTP Responses (inbound):
  - Enabled
    
    default: on
    
    on/off toggle
    - on:
      intercept HTTP Responses and apply its corresponding set of rules
    - off:
      disable this feature entirely
  - Path to Rules File
    
    default: ''
    
    refer to Comments / Implementation Notes for advanced usage
  - Watch Interval (ms, 0 to disable)
    
    default: 0 (off)
    
    useful while writing/testing new rules.
    this feature will watch the rules file for changes, and reload its contents as needed.
output: save()
- HTTP Request Persistence:
  - Enabled
    
    default: on
    
    on/off toggle
    - on:
      the save() helper function will save a record of the request to Output File
    - off:
      disable this feature entirely
  - Path to Output File
    
    default: ''
    
    refer to Comments / Implementation Notes for advanced usage
  - Maximum Number of Saved Requests
    
    default: 10
    
    this feature is intended to prevent the Output File from growing too large
    - > 0:
      when a request record is saved to Output File, the data is prepended. If after this addition there are more records stored in the file (ie: N) than the specified maximum number of records (ie: X), then only the first X are retained… and the trailing (N-X) are removed.
    - 0:
      allow the file to grow without any limitation.
tools to replay saved requests
- common settings:
  - Path to Download Directory
    
    default: {DfltDwnld}
    
    refer to Comments / Implementation Notes for advanced usage
- wget:
  - Path to wget executable
    
    default: /usr/bin/wget
    
    refer to Comments / Implementation Notes for advanced usage
  - Command-Line Options for wget executable
    
    default: -c -nd --content-disposition --no-http-keep-alive --no-check-certificate -e robots=off --progress=dot:binary
- curl:
  
  a reference implementation that adds support for this tool exists in a separate branch: js/eval/replay/curl
  
  this hasn't been merged into the js/eval/master branch due to a small incompatability, which is described pretty well across both:
  - the release notes for: js/eval/v2.00
  - the commit message for: 32cb770

Hidden Preferences

extensions.Moz-Rewrite.debug

default: false

boolean
- true:
  enables debug log messages to be printed to the Browser Console
- false:
  suppresses these log messages
extensions.Moz-Rewrite.case_sensitive

default: false

boolean
- true:
  the alphabetic case of the URL and its components are preserved
- false:
  the URL and its components are always normalized to lowercase, which allows regex patterns to be written in lowercase and omit a case insensitive flag

Dialog Windows

Tools -> moz-rewrite -> user preferences
- same Options dialog as:
  - Tools -> Add-ons
    - Extensions
      - moz-rewrite -> Options
- provides a graphical interface for the user to apply changes to the values of (non-hidden / user) addon preferences
Tools -> moz-rewrite -> view/replay saved requests
- saved HTTP Requests:
  - list of all saved requests.
    for each, a checkbox is followed by the corresponding URL.
- common form field controls
  - replay selected requests using..
    button that displays a list of all supported download tools.
    this list currently contains:
    - wget
  - interactively identify each partial/incomplete download file
    checkbox that
    - when:
      
      one or more saved HTTP Requests are selected
      
      a download tool is chosen/activated from the list
    - if checked:
      
      for each of the selected saved HTTP Requests, an interactive file picker dialog will open and allow the user to choose the file path for the download.
      
      this workflow allows using the external download tool to be used to save data to arbitrary paths within the filesystem,
      rather than only to the Download Directory.
      
      this is particularly useful for when the browser begins a download, but fails to complete.
      in such a case,
      
      if the request was saved…
      or if the browser can re-request the download, and this subsequent request is saved…
      without actually saving the file, and certainly NOT over writing the previously downloaded partial/incomplete file
      
      then the interactive dialog would allow the user to browse for this partial/incomplete file download
    - if not checked:
      
      for each of the selected saved HTTP Requests, the chosen download tool will begin saving/resuming the requested data.
      
      this data will be saved to a file in the Download Directory
      
      the filename will be determined by the external download tool.
      factors that the tool may take into consideration:
      
      a filename component of the requested URL
      
      a 'content-disposition' header of the response
      
      command-line options for the tool (in addon preferences)
  - fallback behavior when 'cancel' is chosen in interactive dialog
    checkbox that
    - when:
      
      interactively identify each partial/incomplete download file is checked
      
      an interactive file picker dialog is closed by the user without having selected a filepath
      
      cancel button
      
      close window (ie: "X") button
    - if checked:
      
      proceed with download and save to default directory
    - if not checked:
      
      skip replay of the specific saved request

Alternate Variation

the moz-rewrite project has several branches.
Each contains an alternate variation that follows a different implementation methodology.
Two of which are intended for distribution here, on AMO:
- (alternate variation)
  moz-rewrite-json requires that the user defined rules data be format in JSON
- (current page)
  moz-rewrite-js allows the user defined rules data to be format in Javascript
  - the headers and stop attributes can be declared as user-defined functions, which are called per request/response and return data that is conditional on its current state
  - the state of each request/response is collected into contextual variables that are available and in-scope while user-defined functions are called
  - a library of helper functions are also available and in-scope while user-defined functions are called
  - this library includes the ability to:
    - cancel() a request
    - redirectTo(url) a request
    - save() a request, which includes all of its HTTP headers and POST data
  - the UI includes a dialog window that displays all saved requests, and provides the ability to replay any subset of saved requests using a (supported) external download utility (ex: wget)
    - resume partial/incomplete file downloads
    - etc…

Developer’s Comments

When posting an issue on github, please indicate the branch of this addon that you are using.

There are currently 3 variations:

user defined data rules are format in JSON, and parsed with JSON.parse()
- AMO: moz-rewrite-json
- github branch: json/master
(current page)
user defined data rules are format in Javascript, and evaluated with Cu.evalInSandbox()
- AMO: moz-rewrite-js
- github branch: js/Cu.evalInSandbox/master
user defined data rules are format in Javascript, and evaluated with eval()
- AMO: N/A
- github branch: js/eval/master

Add-ons

Welcome to Firefox Add-ons.

On the go?

Rewrite HTTP Headers (JS) 1.01.1-signed

by Warren Bank

About this Add-on

Developer’s Comments

Version Information

Version 1.01.1-signed Released May 4, 2015 63.0 kB Works with Firefox 4.0 and later

Add-ons

Welcome to Firefox Add-ons.

On the go?

Rewrite HTTP Headers (JS) 1.01.1-signed

by Warren Bank

About this Add-on

Developer’s Comments

Version Information

Version 1.01.1-signed Released May 4, 2015 63.0 kB Works with Firefox 4.0 and later

Report Abuse