some tweaks before a new release

.gitlab/issue_templates/Bug.md

+Hi,
+
+You are about to submit a bug report.
+
+First of all, make sure you are actually facing a bug.
+If you have some questions about how to setup Burp-UI, make sure you read the
+[doc](https://burp-ui.readthedocs.io/en/latest/) first and especially the
+[FAQ](https://burp-ui.readthedocs.io/en/latest/faq.html) which already answers a
+couple of questions.
+
+Now, if you are sure you are facing a bug, please make sure to provide the
+following informations:
+
+- Bug summary
+- Burp version: `burp -v`
+- Burp-UI version: `burp-ui -V -v`
+- Python version: `python --version`
+- List the steps to reproduce your issue
+- Any log that might help understand/reproduce the problem: `burp-ui -vvvv`
+- Any piece of configuration that might help understand/reproduce the problem
+- Any other information that you may find useful such as screenshots, etc.
+
+Thanks
+
+Below is an example of a expected bug report:
+
+----------------------------------------
+
+Hello,
+
+I have some trouble with Burp-UI right now. Here is a bug report:
+
+# Bug summary
+
+Unable to login: SQL error
+
+# Burp
+
+```
+$ burp -v
+burp-2.0.54
+```
+
+# Burp-UI
+
+```
+$ burp-ui -V -v
+burp-ui: v0.4.0 (stable)
+```
+
+# Python
+
+```
+$ python --version
+Python 3.6.0
+```
+
+# Steps to reproduce
+
+1. Go to the login page
+2. Try to authenticate
+3. Authentication fail with a HTTP 500 Error
+
+# logs
+
+```
+10.0.0.100 - - [11/Apr/2017 15:10:31] "POST /login?next=%2F HTTP/1.1" 500 -
+Traceback (most recent call last):
+  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1994, in __call__
+    return self.wsgi_app(environ, start_response)
+  File "/opt/workspace/burp-ui/burpui/utils.py", line 412, in __call__
+    return self.wsgi_app(environ, start_response)
+  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1985, in wsgi_app
+    response = self.handle_exception(e)
+  File "/usr/local/lib/python2.7/dist-packages/flask_restplus/api.py", line 557, in error_router
+    return original_handler(e)
+  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1540, in handle_exception
+    reraise(exc_type, exc_value, tb)
+  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1982, in wsgi_app
+    response = self.full_dispatch_request()
+  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1614, in full_dispatch_request
+    rv = self.handle_user_exception(e)
+  File "/usr/local/lib/python2.7/dist-packages/flask_restplus/api.py", line 557, in error_router
+    return original_handler(e)
+  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1517, in handle_user_exception
+    reraise(exc_type, exc_value, tb)
+  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1612, in full_dispatch_request
+    rv = self.dispatch_request()
+  File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1598, in dispatch_request
+    return self.view_functions[rule.endpoint](**req.view_args)
+  File "/opt/workspace/burp-ui/burpui/routes.py", line 409, in login
+    user = bui.uhandler.user(form.username.data, refresh)
+  File "/opt/workspace/burp-ui/burpui/misc/auth/handler.py", line 52, in user
+    session_manager.session_expired()
+  File "/opt/workspace/burp-ui/burpui/sessions.py", line 39, in session_expired
+    return self.session_expired_by_id(self.get_session_id())
+  File "/opt/workspace/burp-ui/burpui/sessions.py", line 47, in session_expired_by_id
+    store = Session.query.filter_by(uuid=id).first()
+  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2697, in first
+    ret = list(self[0:1])
+  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2489, in __getitem__
+    return list(res)
+  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2797, in __iter__
+    return self._execute_and_instances(context)
+  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2820, in _execute_and_instances
+    result = conn.execute(querycontext.statement, self._params)
+  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 945, in execute
+    return meth(self, multiparams, params)
+  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
+    return connection._execute_clauseelement(self, multiparams, params)
+  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
+    compiled_sql, distilled_params
+  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
+    context)
+  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1393, in _handle_dbapi_exception
+    exc_info
+  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
+    reraise(type(exception), exception, tb=exc_tb, cause=cause)
+  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
+    context)
+  File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 469, in do_execute
+    cursor.execute(statement, parameters)
+OperationalError: (sqlite3.OperationalError) no such table: session [SQL: u'SELECT session.id AS session_id, session.uuid AS session_uuid, session.user AS session_user, session.ip AS session_ip, session.ua AS session_ua, session.timestamp AS session_timestamp, session.expire AS session_expire, session.permanent AS session_permanent, session.api AS session_api \nFROM session \nWHERE session.uuid = ?\n LIMIT ? OFFSET ?'] [parameters: (u'ae350427-99f4-4592-94ec-6f6a27aee59f', 1, 0)]
+```
+
+# Configuration
+
+```
+[Global]
+# burp server version 1 or 2
+version = 1
+# Handle multiple bui-servers or not
+# If set to 'false', you will need to declare at least one 'Agent' section (see
+# bellow)
+single = false
+# authentication plugin (mandatory)
+# list the misc/auth directory to see the available backends
+# to disable authentication you can set "auth: none"
+# you can also chain multiple backends. Example: "auth: ldap,basic"
+# the order will be respected unless you manually set a higher backend priority
+auth = basic, ldap
+# acl plugin
+# list misc/auth directory to see the available backends
+# default is no ACL
+acl = basic
+# You can change the prefix if you are behind a reverse-proxy under a custom
+# root path. For example: /burpui
+prefix = none
+
+[Production]
+# storage backend (only used with gunicorn) for session and cache
+# may be either 'default' or 'redis'
+storage = redis
+# session database to use
+# may also be a backend url like: redis://localhost:6379/0
+# if set to 'redis', the backend url defaults to:
+# redis://<redis_host>:<redis_port>/0
+# where <redis_host> is the host part, and <redis_port> is the port part of
+# the below "redis" setting
+session = redis
+# cache database to use
+# may also be a backend url like: redis://localhost:6379/0
+# if set to 'redis', the backend url defaults to:
+# redis://<redis_host>:<redis_port>/1
+# where <redis_host> is the host part, and <redis_port> is the port part of
+# the below "redis" setting
+cache = redis
+# redis server to connect to
+redis = localhost:6379
+# whether to use celery
+celery = true
+# database url to store some persistent data
+# example: sqlite:////var/lib/burpui/store.db
+database = sqlite:////tmp/burpui.db
+```
+
+Thanks

.gitlab/issue_templates/Feature Request.md

+Hi,
+
+You are about to submit a ~"feature request".
+
+Here are a couple of rules to follow in order to get your request approved:
+
+- Be polite
+- Provide an accurate description of what you expect
+- Don't forget to add the ~"feature request" label
+- Keep in mind I work on Burp-UI on my spare time so it may take some time to
+  get your feature request implemented
+
+Thanks

burpui/app.py

@@ -130,7 +130,7 @@ def create_db(myapp, cli=False, unittest=False, create=True):
         except ImportError:  # pragma: no cover
             myapp.logger.critical(
                 'Unable to load requirements, you may want to run \'pip '
-                'install burp-ui-sql\'.\nDisabling SQL support for now.'
+                'install "burp-ui[sql]"\'.\nDisabling SQL support for now.'
             )
             myapp.config['WITH_SQL'] = False
         except OperationalError as exp:  # pragma: no cover

burpui/cli.py

@@ -290,6 +290,7 @@ def setup_burp(bconfcli, bconfsrv, client, host, redis, database, dry):
             # detect missing modules
             import redis as redis_client  # noqa
             import celery  # noqa
+            import socket
             if ('redis' not in app.conf.options['Production'] or
                 'redis' in app.conf.options['Production'] and
                 app.conf.options['Production']['redis'] != redis) and \
@@ -297,8 +298,19 @@ def setup_burp(bconfcli, bconfsrv, client, host, redis, database, dry):
                 app.conf.options['Production']['redis'] = redis
 
             rhost, rport, _ = get_redis_server(app)
-            DEVNULL = open(os.devnull, 'wb')
-            ret = subprocess.call(['/bin/nc', '-z', '-w5', str(rhost), str(rport)], stdout=DEVNULL, stderr=subprocess.STDOUT)
+            ret = -1
+            for res in socket.getaddrinfo(rhost, rport, socket.AF_UNSPEC, socket.SOCK_STREAM):
+                if ret == 0:
+                    break
+                af, socktype, proto, _, sa = res
+                try:
+                    s = socket.socket(af, socktype, proto)
+                except socket.error:
+                    continue
+                try:
+                    ret = s.connect_ex(sa)
+                except:
+                    continue
 
             if ret == 0:
                 app.conf.options['Production']['celery'] = 'true'
@@ -344,7 +356,7 @@ def setup_burp(bconfcli, bconfsrv, client, host, redis, database, dry):
             click.echo(
                 click.style(
                     'It looks like some dependencies are missing. Did you ran '
-                    'the \'pip install burp-ui-sql\' command first?',
+                    'the \'pip install "burp-ui[sql]"\' command first?',
                     fg='yellow'
                 )
             )

burpui/templates/gerard.js

@@ -217,28 +217,33 @@ var substringMatcher = function(objs) {
 var _clients_all = [];
 
 	{% if config.STANDALONE -%}
+
 $.get("{{ url_for('api.clients_all') }}")
 	.done(function (data) {
 		_clients_all = data;
+
+		/***
+		 * Map out _clients_bh to our input with the typeahead plugin
+		 */
+		$('#input-client').typeahead({
+			highlight: true
+		},
+		{
+			name: 'clients',
+			displayKey: 'name',
+			source: substringMatcher(_clients_all),
+		}).on('typeahead:selected', function(obj, datum, name) {
+			window.location = '{{ url_for("view.client") }}?name='+datum.name;
+		});
+
 	});
 
-/***
- * Map out _clients_bh to our input with the typeahead plugin
- */
-$('#input-client').typeahead({
-	highlight: true
-},
-{
-	name: 'clients',
-	displayKey: 'name',
-	source: substringMatcher(_clients_all),
-}).on('typeahead:selected', function(obj, datum, name) {
-	window.location = '{{ url_for("view.client") }}?name='+datum.name;
-});
 	{% else -%}
 
 		{% for srv in config.SERVERS -%}
+
 var _clients_{{ srv }} = [];
+
 		{% endfor -%}
 
 $.get("{{ url_for('api.clients_all') }}")

docs/buiagent.rst

@@ -80,9 +80,11 @@ following command:
 
 ::
 
-    pip install "burp-ui[agent]"
+    pip install burp-ui-agent
 
 
+.. note:: Starting with *v0.5.0*, the agent now has its own package.
+
 Configuration
 -------------
 
@@ -128,27 +130,6 @@ As with `Burp-UI`_, you need a specific section depending on the *version*
 value. Please refer to the `Burp-UI versions <advanced_usage.html#versions>`__
 section for more details.
 
-Daemon
-------
-
-I have no plan to implement daemon features, but there are a lot of tools
-available to help you achieve such a behavior.
-
-For instance, you can create a systemd service file containing:
-
-::
-
-    [Unit]
-    Description=Burp-UI agent service
-    After=network.target
-
-    [Service]
-    ExecStart=/usr/local/bin/bui-agent
-    User=burpui
-
-
-You can also have a look at how the demo works (it uses supervisor)
-
 Example
 -------
 
@@ -173,7 +154,18 @@ http://front:5000/ for instance, and the `Burp-UI`_ instance (front) will
 Service
 =======
 
-To run bui-agent as service. 
+I have no plan to implement daemon features, but there are a lot of tools
+available to help you achieve such a behavior.
+
+To run bui-agent as a service, a systemd file is provided. You can use it like
+this:
+
+::
+
+    cp /usr/local/share/burpui/contrib/systemd/bui-agent.service /etc/systemd/system/
+    systemctl daemon-reload
+    systemctl enable bui-agent.service
+    systemctl start bui-agent.service
 
 
 

docs/contributing.rst

@@ -143,7 +143,7 @@ Optionally your python version and your OS might be useful as well.
 Questions
 ---------
 
-Ask questions in the `discussion forum <https://burpui.ziirish.me/>`__. Do not
+Ask questions in the `discussion forum <https://forum.burp-ui.org/>`__. Do not
 use the issue tracker for this purpose.
 
 ``Burp-UI`` has extensive online documentation please read the `doc

docs/manage.rst

@@ -80,7 +80,7 @@ You will also need some extra requirements:
 
 ::
 
-    pip install --upgrade burp-ui-sql
+    pip install --upgrade "burp-ui[sql]"
 
 
 Then you just have to run the following command to have your database setup:

docs/requirements.rst

@@ -77,7 +77,7 @@ If you need persistent data, you will need additional dependencies as well:
 
 ::
 
-    pip install burp-ui-sql
+    pip install "burp-ui[sql]"
 
 
 Now if you want to use a MySQL database, you will need the proper driver. For

docs/upgrading.rst

@@ -53,7 +53,7 @@ v0.4.0
 - **New** - The ``bui-manage`` tool can now help you setup both `Burp`_ and
   `Burp-UI`_.
 - **New** - The SQL requirements have evolved, you **MUST** run
-  ``pip install --upgrade burp-ui-sql`` if you wish to keep using persistent
+  ``pip install --upgrade "burp-ui[sql]"`` if you wish to keep using persistent
   storage.
 
 

setup.py

@@ -5,7 +5,7 @@ import os
 import re
 import sys
 
-from subprocess import check_output, call
+from subprocess import check_output, call, STDOUT
 from distutils import log
 from distutils.core import Command
 from setuptools import setup, find_packages
@@ -16,6 +16,7 @@ from setuptools.command.bdist_egg import bdist_egg
 from setuptools.command.egg_info import egg_info
 
 ROOT=os.path.join(os.path.dirname(os.path.realpath(__file__)))
+DEVNULL = open(os.devnull, 'wb')
 
 # Not sure bower was a great idea...
 VENDOR_TO_KEEP = [
@@ -140,21 +141,21 @@ class BuildStatic(Command):
     def run(self):
         os.chdir(ROOT)
         log.info("getting revision number")
-        call('{} ./burpui -m manage compile_translation'.format(sys.executable).split())
+        call('{} ./burpui -m manage compile_translation'.format(sys.executable).split(), stderr=DEVNULL)
         rev = 'stable'
-        if os.path.exists('.git/HEAD'):
+        if os.path.exists('.git') and call("which git", shell=True, stderr=STDOUT, stdout=DEVNULL) == 0:
             try:
-                branch = check_output('sed s@^.*/@@g .git/HEAD'.split()).rstrip()
+                branch = check_output('git rev-parse HEAD', shell=True).rstrip()
                 ver = open(os.path.join('burpui', 'VERSION')).read().rstrip()
                 if branch and 'dev' in ver:
                     rev = branch
-                try:
-                    with open('burpui/RELEASE', 'w') as f:
-                        f.write(rev)
-                except:
-                    pass
             except:
                 pass
+        try:
+            with open('burpui/RELEASE', 'w') as f:
+                f.write(rev)
+        except:
+            pass
         keep = VENDOR_TO_KEEP
         dirlist = []
         for dirname, subdirs, files in os.walk('burpui/static/vendor'):