burp-ui cache issues + remove gevent calls

Hi,

You told me to talk to you again about caching issues if I see some.

Updated to latest v0.7.0-dev from today. I had my bui-agent service that didn't want to run in the first place (some stupid error I made), so I had Burp-UI not showing my servers as Status 'X'.

Fixing my bui-agent service file, my session still shows now my server status as 'X', it's been a some minutes.

[EDIT] After having finished writing this issue, I updated my browser window, and everything seems to work now... Cache expired ?[/EDIT]

Opening my browser in incognito mode, I was able to click on the server (which status was fine) and browse among backups until I tried to open a particular backup which gave me the following: cannot switch to a different thread

bui-agent corresponding log:

[2019-07-30 09:30:40,184] INFO in agent.handle: recv: b'{"func": "get_all_clients", "args": {}, "password": "somepass"}'
[2019-07-30 09:30:40,185] INFO in burp2.status: 140300300577704 - query: 'c:' (cache: True, raw: False)
[2019-07-30 09:30:40,186] INFO in agent.handle: recv: b'{"func": "get_all_clients", "args": {}, "password": "somepass"}'
[2019-07-30 09:30:40,187] ERROR in agent.handle: cannot switch to a different thread
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/burpui_agent/engines/agent.py", line 293, in handle
    res = json.dumps(await trio.run_sync_in_worker_thread(callback))
  File "/usr/local/lib/python3.6/site-packages/burpui_agent/../trio/_threads.py", line 400, in run_sync_in_worker_thread
    return await _core.wait_task_rescheduled(abort)
  File "/usr/local/lib/python3.6/site-packages/burpui_agent/../trio/_core/_traps.py", line 166, in wait_task_rescheduled
    return (await _async_yield(WaitTaskRescheduled(abort_func))).unwrap()
  File "/usr/local/lib/python3.6/site-packages/burpui_agent/../outcome/_sync.py", line 111, in unwrap
    raise captured_error
  File "/usr/local/lib/python3.6/site-packages/burpui_agent/../trio/_threads.py", line 362, in do_release_then_return_result
    return result.unwrap()
  File "/usr/local/lib/python3.6/site-packages/burpui_agent/../outcome/_sync.py", line 111, in unwrap
    raise captured_error
  File "/usr/local/lib/python3.6/site-packages/burpui_agent/misc/backend/burp2.py", line 604, in get_all_clients
    query = self.status()
  File "/usr/local/lib/python3.6/site-packages/burpui_agent/misc/backend/burp2.py", line 137, in status
    with self.plock:
  File "/usr/local/lib64/python3.6/site-packages/gevent/lock.py", line 230, in __enter__
    return self.acquire()
  File "/usr/local/lib64/python3.6/site-packages/gevent/lock.py", line 223, in acquire
    rc = self._block.acquire(blocking)
  File "src/gevent/_semaphore.py", line 100, in gevent.__semaphore.Semaphore.acquire
  File "src/gevent/_semaphore.py", line 128, in gevent.__semaphore.Semaphore.acquire
  File "src/gevent/_abstract_linkable.py", line 192, in gevent.__abstract_linkable.AbstractLinkable._wait
  File "src/gevent/_abstract_linkable.py", line 165, in gevent.__abstract_linkable.AbstractLinkable._wait_core
  File "src/gevent/_abstract_linkable.py", line 169, in gevent.__abstract_linkable.AbstractLinkable._wait_core
  File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 60, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 64, in gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/__greenlet_primitives.pxd", line 35, in gevent.__greenlet_primitives._greenlet_switch
greenlet.error: cannot switch to a different thread
[2019-07-30 09:30:40,188] WARNING in agent.handle: Forwarding Exception: cannot switch to a different thread

Using another incognito window, everything works fine and I can browse the same backup. So I guess there's still some caching issues, cannot diagnose what they are and where they come from though. My best guess is that the cache isn't validated against changes.

Burp

$ burp -v
burp-2.3.6

Sysinfo

$ bui-manage sysinfo
/usr/local/share/burpui/migrations
{'sqlalchemy': <flask_sqlalchemy._SQLAlchemyState object at 0x7f3d8656a7b8>}
Python version:      3.6.8
Burp-UI version:     0.7.0.dev0 ()
OS:                  Linux:3.10.0-957.21.3.el7.x86_64 (posix)
Distribution:        centos 7.6.1810 Core
Single mode:         False
Backend:             multi
WebSocket embedded:  False
WebSocket available: True
Config file:         /etc/burp/burpui.cfg

Steps to reproduce (hopefully)

Have an agent setup
Disable agent service
Login and see server status 'x'
Enable agent service again
Update view, there are changes server status remains 'x'

Configuration

[Global]
backend = multi
auth = basic
acl = basic
audit = basic
plugins = none

[UI]
refresh = 180
liverefresh = 5
ignore_labels = color:.*, custom:.*
format_labels = s/^os:\s*//
default_strip = 0

[Production]
storage = redis
session = redis
cache = redis
redis = localhost:6379
celery = true
database = sqlite:////var/lib/burpui/store.db
limiter = false
ratio = 60/minute
prefix = none
num_proxies = 0
proxy_fix_args = "{'x_for': {num_proxies}, 'x_host': {num_proxies}, 'x_prefix': {num_proxies}}"

[WebSocket]
enabled = false
embedded = false
broker = redis
url = none
debug = false

[Security]
includes = /etc/burp
enforce = false
revoke = true
cookietime = 1
sessiontime = 1
scookie = true
appsecret = somestring

[Experimental]
zip64 = false
noserverrestore = false

[Burp]
burpbin = /usr/sbin/burp
stripbin = /usr/sbin/vss_strip
bconfcli = /etc/burp/burp.conf
bconfsrv = /etc/burp/burp-server.conf
tmpdir = /tmp/bui
timeout = 15
deep_inspection = false

[Parallel]
host = ::1
port = 11111
timeout = 15
password = password123456
ssl = true
concurrency = 2
init_wait = 15

[BASIC:AUDIT]
priority = 100
level = WARNING
logfile = /var/log/burp-ui.log
max_bytes = 30 * 1024 * 1024
rotate = 5


[BASIC:AUTH]
priority = 100
admin = pbkdf2:sha256:somesalt



[ACL]
extended = true
assume_rw = true
legacy = false

[BASIC:ACL]
priority = 100
admin = admin
supadmin = '{"agents":{"Agent1":{"rw":["client.*","server.*"]},"Agent2":{"rw":["client.*","server.*"]}}}'



[Agent:Agent1]
host = 127.0.0.1
port = 10000
password = somepass
ssl = false
timeout = 300

[Agent:Agent1]
host = 127.0.0.1
port = 10001
password = somepass
ssl = false
timeout = 300

Thanks

Edited Jul 30, 2019 by Benjamin "Ziirish" SANS