Discussion:
[Reportbug-maint] Bug#892898: reportbug crashes due with UnicodeDecodeError
Mikulas Patocka
2018-03-14 08:21:25 UTC
Permalink
Package: reportbug
Version: 7.1.10
Severity: normal

Dear Maintainer,

Reportbug crashes when trying to report a bug for the package 'console-setup'.
The crash happens in text, urwid and gtk2 environments.
The crash happens on Stretch and Sid, it doesn't happen on Jessie.
The crash happens with locales cs_CZ.utf8 and cs_CZ.iso8859-2

*** Reporter, please consider answering these questions, where appropriate ***

* What led up to the situation?

I tried to report bug for the package console-setup.

* What exactly did you do (or not do) that was effective (or
ineffective)?

Start "reportbug -u urwid"
Type "console-setup"
A list of bugs for the package console-setup appears.
Select "New bug"

* What was the outcome of this action?

Reportbug crashes with this error:
Note: bug reports are publicly archived (including the email address of the submitter).
Detected character set: UTF-8
Please change your locale if this is incorrect.

Using 'Mikulas Patocka <***@twibright.com>' as your from address.
Getting status for console-setup...
Will send report to Debian (per lsb_release).
Maintainer for console-setup is 'Debian Install System Team <debian-***@lists.debian.org>'.
Looking up dependencies of console-setup...
Getting status for related package ...
Getting status for related package keyboard-configuration...
Looking up 'depends' of related package keyboard-configuration...
Getting status for related package console-setup-mini...
Getting status for related package console-setup-linux...
Looking up 'depends' of related package console-setup-linux...
Looking up 'suggests' of related package console-setup-linux...
Getting status for related package console-setup-freebsd...
Looking up status of additional packages...
Traceback (most recent call last):
File "/usr/bin/reportbug", line 2265, in <module>
main()
File "/usr/bin/reportbug", line 1109, in main
return iface.user_interface()
File "/usr/bin/reportbug", line 1849, in user_interface
'/usr/bin/debconf-show %s' % showpkgs)
File "/usr/lib/python3.6/subprocess.py", line 517, in getstatusoutput
data = check_output(cmd, shell=True, universal_newlines=True, stderr=STDOUT)
File "/usr/lib/python3.6/subprocess.py", line 336, in check_output
**kwargs).stdout
File "/usr/lib/python3.6/subprocess.py", line 405, in run
stdout, stderr = process.communicate(input, timeout=timeout)
File "/usr/lib/python3.6/subprocess.py", line 830, in communicate
stdout = self.stdout.read()
File "/usr/lib/python3.6/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 583: invalid continuation byte

Note that when we try it with the locale cs_CZ.iso8859-2, we get a slightly different error:
Traceback (most recent call last):
File "/usr/bin/reportbug", line 2265, in <module>
main()
File "/usr/bin/reportbug", line 1109, in main
return iface.user_interface()
File "/usr/bin/reportbug", line 1856, in user_interface
"Don't send your debconf settings.", nowrap=True)):
File "/usr/lib/python3/dist-packages/reportbug/ui/urwid_ui.py", line 362, in yes_no
result = box.main(ui)
File "/usr/lib/python3/dist-packages/reportbug/ui/urwid_ui.py", line 206, in main
return self.ui.run_wrapper(self.run)
File "/usr/lib/python3/dist-packages/urwid/display_common.py", line 773, in run_wrapper
return fn()
File "/usr/lib/python3/dist-packages/reportbug/ui/urwid_ui.py", line 158, in run
self.ui.draw_screen(size, canvas)
File "/usr/lib/python3/dist-packages/urwid/raw_display.py", line 850, in draw_screen
self.write(l)
File "/usr/lib/python3/dist-packages/urwid/raw_display.py", line 279, in write
self._term_output_file.write(data)
File "/usr/lib/python3.6/encodings/iso8859_2.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\ufffd' in position 37: character maps to <undefined>


* What outcome did you expect instead?

The python script shouldn't crash due to encoding errors.

*** End of the template - remove these template lines ***


-- Package-specific info:
** Environment settings:
INTERFACE="urwid"

** /home/mikulas/.reportbugrc:
reportbug_version "6.6.6"
mode standard
ui text
email "***@twibright.com"
smtphost "leontynka"

-- System Information:
Debian Release: buster/sid
APT prefers unreleased
APT policy: (500, 'unreleased'), (500, 'unstable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386, x32

Kernel: Linux 4.16.0-rc5 (SMP w/12 CPU cores; PREEMPT)
Locale: LANG=cs_CZ.utf8, LC_CTYPE=cs_CZ.utf8 (charmap=UTF-8), LANGUAGE=cs_CZ.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: sysvinit (via /sbin/init)

Versions of packages reportbug depends on:
ii apt 1.6~beta1
ii python3 3.6.4-1
ii python3-reportbug 7.1.10
ii sensible-utils 0.0.12

reportbug recommends no packages.

Versions of packages reportbug suggests:
pn claws-mail <none>
pn debconf-utils <none>
pn debsums <none>
pn dlocate <none>
pn emacs24-bin-common | emacs25-bin-common <none>
ii file 1:5.32-2
pn gir1.2-gtk-3.0 <none>
pn gir1.2-vte-2.91 <none>
pn gnupg | pgp <none>
pn postfix | exim4 | mail-transport-agent <none>
pn python3-gi <none>
pn python3-gi-cairo <none>
pn python3-gtkspellcheck <none>
ii python3-urwid 2.0.1-1
pn xdg-utils <none>

Versions of packages python3-reportbug depends on:
ii apt 1.6~beta1
ii file 1:5.32-2
ii python3 3.6.4-1
ii python3-apt 1.4.0~beta3+b1
ii python3-debian 0.1.32
ii python3-debianbts 2.7.2
ii python3-requests 2.18.4-2

python3-reportbug suggests no packages.

-- no debconf information
Debian Bug Tracking System
2018-03-17 21:33:04 UTC
Permalink
tags -1 patch
Bug #892898 [reportbug] reportbug crashes due with UnicodeDecodeError
Added tag(s) patch.
--
892898: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=892898
Debian Bug Tracking System
Contact ***@bugs.debian.org with problems
Mikulas Patocka
2018-03-19 19:32:11 UTC
Permalink
control: tags -1 patch
https://salsa.debian.org/reportbug-team/reportbug/merge_requests/2
I could not reproduce the problem, though. What does
DEBCONF_SYSTEMRC=1 DEBCONF_NOWARNINGS=yes debconf-show console-setup
output on your system? Is the output not in a consistent encoding?
On Debian-Stretch:
$ DEBCONF_SYSTEMRC=1 DEBCONF_NOWARNINGS=yes debconf-show console-setup
console-setup/fontsize: 11x22
console-setup/codeset47: # Latin2 - central Europe and Romanian
console-setup/fontface47: TerminusBold
console-setup/charmap47: ISO-8859-2
console-setup/fontsize-fb47: 11x22 (pouze framebuffer)
console-setup/use_system_font:
console-setup/store_defaults_in_debconf_db: true
console-setup/codesetcode: Lat2
console-setup/guess_font:
console-setup/fontsize-text47: 11x22 (pouze framebuffer)
console-setup/framebuffer_only:

On Debian-Sid:
$ DEBCONF_SYSTEMRC=1 DEBCONF_NOWARNINGS=yes debconf-show console-setup
console-setup/guess_font:
console-setup/codesetcode: Lat2
console-setup/framebuffer_only:
console-setup/fontsize-text47: 10x20 (pouze framebuffer)
console-setup/fontsize-fb47: 10x20 (pouze framebuffer)
console-setup/charmap47: UTF-8
console-setup/use_system_font:
console-setup/fontface47: TerminusBold
console-setup/codeset47: # Latin2 - central Europe and Romanian
console-setup/store_defaults_in_debconf_db: true
console-setup/fontsize: 10x20

On Debian-Jessie (the bug also shows up in Jessie, I was wrong in the report):
$ DEBCONF_SYSTEMRC=1 DEBCONF_NOWARNINGS=yes debconf-show console-setup
console-setup/fontsize-fb47: 11x22 (pouze framebuffer)
console-setup/fontface47: TerminusBold
console-setup/use_system_font:
console-setup/fontsize-text47: 11x22 (pouze framebuffer)
console-setup/guess_font:
console-setup/charmap47: ISO-8859-2
console-setup/framebuffer_only:
console-setup/store_defaults_in_debconf_db: true
console-setup/codeset47: # Latin2 - central Europe and Romanian
console-setup/fontsize: 11x22
console-setup/codesetcode: Lat2

On Debian Wheezy:
$ DEBCONF_SYSTEMRC=1 DEBCONF_NOWARNINGS=yes debconf-show console-setup
console-setup/codeset47: # Latin2 - central Europe and Romanian
console-setup/framebuffer_only:
console-setup/guess_font:
console-setup/store_defaults_in_debconf_db: true
console-setup/use_system_font:
console-setup/fontsize: 8x16
console-setup/fontface47: TerminusBold
console-setup/fontsize-text47: 8x16
console-setup/charmap47: ISO-8859-2
console-setup/codesetcode: Lat2
console-setup/fontsize-fb47: 8x16



But I think the problem is with the 'keyboard-configuration' package -
reportbug also queries it when reporting a bug for console-setup and it
shows some non-ascii characters. The resulting encoding is independent on
the LANG variable.

On Debian-Stretch:
$ DEBCONF_SYSTEMRC=1 DEBCONF_NOWARNINGS=yes /usr/bin/debconf-show keyboard-configuration
keyboard-configuration/variant: Anglické (USA) <--- iso-8859-2
keyboard-configuration/xkb-keymap: us
keyboard-configuration/model: Obecné PC 105klávesové (mez.) <--- iso-8859-2
keyboard-configuration/layoutcode: us
keyboard-configuration/variantcode:
keyboard-configuration/optionscode:
keyboard-configuration/unsupported_config_options: true
keyboard-configuration/store_defaults_in_debconf_db: true
keyboard-configuration/altgr: The default for the keyboard layout
debian-installer/console-setup-udeb/title:
keyboard-configuration/modelcode: pc105
keyboard-configuration/layout:
keyboard-configuration/other:
keyboard-configuration/compose: No compose key
keyboard-configuration/unsupported_layout: true
keyboard-configuration/unsupported_options: true
keyboard-configuration/ctrl_alt_bksp: false
keyboard-configuration/switch: No temporary switch
keyboard-configuration/unsupported_config_layout: true
keyboard-configuration/toggle: No toggling

On Debian-Sid:
$ DEBCONF_SYSTEMRC=1 DEBCONF_NOWARNINGS=yes /usr/bin/debconf-show keyboard-configuration
keyboard-configuration/ctrl_alt_bksp: false
* keyboard-configuration/modelcode: pc104
* keyboard-configuration/toggle: No toggling
* keyboard-configuration/altgr: The default for the keyboard layout
keyboard-configuration/unsupported_config_layout: true
* keyboard-configuration/store_defaults_in_debconf_db: true
debian-installer/console-setup-udeb/title:
* keyboard-configuration/layoutcode: us
* keyboard-configuration/xkb-keymap: us
* keyboard-configuration/other:
keyboard-configuration/unsupported_layout: true
* keyboard-configuration/model: Obecné PC 104klávesové <--- iso-8859-2
keyboard-configuration/unsupported_config_options: true
* keyboard-configuration/optionscode:
keyboard-configuration/unsupported_options: true
* keyboard-configuration/switch: No temporary switch
* keyboard-configuration/variant: Anglické (USA) <--- iso-8859-2
* keyboard-configuration/layout:
* keyboard-configuration/variantcode:
* keyboard-configuration/compose: No compose key

On Debian-Jessie:
$ DEBCONF_SYSTEMRC=1 DEBCONF_NOWARNINGS=yes /usr/bin/debconf-show keyboard-configuration
keyboard-configuration/unsupported_config_layout: true
* keyboard-configuration/model: ObecnĂŠ PC 105klĂĄvesovĂŠ (mez.) <--- utf-8
* keyboard-configuration/switch: No temporary switch
* keyboard-configuration/optionscode:
keyboard-configuration/unsupported_config_options: true
* keyboard-configuration/other:
* keyboard-configuration/compose: No compose key
* keyboard-configuration/toggle: No toggling
debian-installer/console-setup-udeb/title:
* keyboard-configuration/xkb-keymap: us
* keyboard-configuration/altgr: The default for the keyboard layout
* keyboard-configuration/modelcode: pc105
keyboard-configuration/unsupported_options: true
* keyboard-configuration/variant: AnglickĂŠ (USA) <--- utf-8
keyboard-configuration/unsupported_layout: true
* keyboard-configuration/layout:
keyboard-configuration/ctrl_alt_bksp: false
* keyboard-configuration/store_defaults_in_debconf_db: true
* keyboard-configuration/layoutcode: us
* keyboard-configuration/variantcode:

On Debian-Wheezy:
$ DEBCONF_SYSTEMRC=1 DEBCONF_NOWARNINGS=yes /usr/bin/debconf-show keyboard-configuration
keyboard-configuration/layout:
keyboard-configuration/xkb-keymap: us
keyboard-configuration/variant: Anglické (USA) <--- iso-8859-2
debian-installer/console-setup-udeb/title:
keyboard-configuration/unsupported_options: true
keyboard-configuration/ctrl_alt_bksp: false
keyboard-configuration/variantcode:
keyboard-configuration/other:
keyboard-configuration/optionscode:
keyboard-configuration/modelcode: pc105
keyboard-configuration/unsupported_config_options: true
keyboard-configuration/unsupported_config_layout: true
keyboard-configuration/toggle: No toggling
keyboard-configuration/compose: No compose key
keyboard-configuration/switch: No temporary switch
keyboard-configuration/altgr: The default for the keyboard layout
keyboard-configuration/unsupported_layout: true
keyboard-configuration/model: Obecné PC 105klávesové (mez.) <--- iso-8859-2
keyboard-configuration/layoutcode: us
keyboard-configuration/store_defaults_in_debconf_db: true

Mikulas

Loading...