Update readme. Refactor toldg a little. Provide example files for testing by the user.
parent
3138be8d17
commit
367c38592b
|
@ -1,5 +1,6 @@
|
||||||
# Ignore sensitive data
|
# Ignore example output file
|
||||||
gather.json
|
example/processed
|
||||||
|
example/mappings/unmatched.csv
|
||||||
# ---> Python
|
# ---> Python
|
||||||
# Byte-compiled / optimized / DLL files
|
# Byte-compiled / optimized / DLL files
|
||||||
__pycache__/
|
__pycache__/
|
||||||
|
|
93
README.md
93
README.md
|
@ -1,28 +1,87 @@
|
||||||
# ledgerpy
|
# ledgerpy
|
||||||
|
|
||||||
Scripts to transform different input formats (CSV and OFX) into ledger
|
Scripts to transform different input formats (CSV and OFX) into ledger
|
||||||
accounting files. Includes mapping language to update transaction details
|
accounting files. The scripts allow manipulating ledger transactions based on
|
||||||
automatically.
|
CSV mapping files.
|
||||||
|
|
||||||
There are other [scripts](https://github.com/ledger/ledger/wiki/CSV-Import) that
|
Other [scripts](https://github.com/ledger/ledger/wiki/CSV-Import) attempt to
|
||||||
attempt to handle the same use-cases. I have tried a couple of them, as well as
|
handle the same use-case. I have tried a couple of them, as well as the
|
||||||
hledger's integrated CSV import, and ran into issues or didn't like the
|
integrated CSV import of hledger, and ran into issues with all of them. That's
|
||||||
usability. That's why I wrote my own scripts for my workflow. Probably not too
|
why I wrote yet another CSV to ledger tool.
|
||||||
useful for anybody else, but I included an example workspace to showcase how I
|
|
||||||
use the scripts.
|
There are two scripts, getofx, and toldg. The former uses the Python
|
||||||
|
[ofxtools](https://ofxtools.readthedocs.io/en/latest/) library to download bank
|
||||||
|
transactions via OFX and stores them into CSV files. The latter takes CSV files
|
||||||
|
and transforms them into ledger accounting files.
|
||||||
|
|
||||||
|
The OFX script works well for my workflow, but I am not sure if it would be
|
||||||
|
beneficial to other people. My premise is that I update my accounting files at
|
||||||
|
least once a week. Therefore, the OFX script downloads the instructions for the
|
||||||
|
last thirty days and then merges them into the CSV file (if it exists).
|
||||||
|
|
||||||
|
You might object that it makes more sense to download all the data for the first
|
||||||
|
run and then update for consecutive runs. That was my idea in the beginning, but
|
||||||
|
it turns out that the OFX credit card interface of my bank only returns the
|
||||||
|
transactions of the last 60 days. Hence, I downloaded all transactions manually
|
||||||
|
and then set up in the incremental updates. The examples directory contains an
|
||||||
|
example configuration for this script.
|
||||||
|
|
||||||
|
On the other hand, I am pretty happy with the CSV transformation script. Most of
|
||||||
|
my workflows are editor based, and the mapping file-based approach makes it easy
|
||||||
|
to manipulate transactions. Also, the script relies on a single configuration
|
||||||
|
file, which makes the configuration clearer.
|
||||||
|
|
||||||
## Dependencies
|
## Dependencies
|
||||||
|
|
||||||
- jinja2
|
The scripts rely on a couple of newer Python features, such as data-classes,
|
||||||
- ofxtools
|
format-strings, and the walrus-operator. Python 3.8 or later is therefore
|
||||||
- python3.8 or higher
|
required to run the scripts. Additionally, the OFX script relies on ofxtools
|
||||||
|
(`pip install ofxtools`). Transforming CSV files into the ledger format does not
|
||||||
|
require additional packages.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
Invoke python3.8 with either of the scripts and provide a configuration file as
|
||||||
|
the first argument.
|
||||||
|
|
||||||
|
```
|
||||||
|
git clone https://git.felixm.de/felixm/ledgerpy.git
|
||||||
|
cd ledgerpy/example
|
||||||
|
python3 ../toldg.py configs/toldg.json # or `make` alternatively
|
||||||
|
```
|
||||||
|
|
||||||
|
You can see that toldg copies and transforms the input files in `processed`
|
||||||
|
directory using the mappings from the CSV files in the `mappings` directory.
|
||||||
|
For transactions that do not have a matching mapping toldg creates a default
|
||||||
|
mapping into `mappings/unmatched.csv`, as shown in the following listing.
|
||||||
|
|
||||||
|
```
|
||||||
|
expenses,UBER ABO13,credit=-29.37;date=2018/12/03
|
||||||
|
```
|
||||||
|
|
||||||
|
The first part is the new payee (or account2) for the transaction. The second
|
||||||
|
part is the string used to match the description of the transaction. Normally,
|
||||||
|
the script uses a string compare. If the string starts and ends with a
|
||||||
|
frontslash `/` toldg compiles it into a regex and tries to match the
|
||||||
|
description: `regex.match(descriptio)`. The last field is a query specification
|
||||||
|
string in the following form:
|
||||||
|
|
||||||
|
```
|
||||||
|
field1=string1;field2=string2
|
||||||
|
```
|
||||||
|
|
||||||
|
I have added this feature to specify different payees for the same store. For
|
||||||
|
example, sometimes a get groceries from Target and other times furniture
|
||||||
|
(household expenses). In case multiple mappings match a transaction the script
|
||||||
|
uses the first match. I might change that into the mapping that has the most
|
||||||
|
query parameters. That way one could have one default mapping (usually I get
|
||||||
|
groceries from Target), but then override on a case per case basis without
|
||||||
|
getting warnings.
|
||||||
|
|
||||||
## Todo
|
## Todo
|
||||||
|
|
||||||
- [ ] Write this readme
|
- [x] Write this readme
|
||||||
- [ ] Create setup.py file
|
- [x] Use OFX parser from ofxtools instead of parsing the XML
|
||||||
- [ ] Use OFX parser from ofxtools instead of parsing the XML
|
- [x] Autoappend latest OFX data to CSV file
|
||||||
- [ ] Autoappend latest OFX data to CSV file
|
- [x] Include example workspace with mock data to demo my workflow
|
||||||
- [ ] Include example workspace with mock data to demo my workflow
|
|
||||||
|
|
||||||
|
|
||||||
|
|
|
@ -0,0 +1,44 @@
|
||||||
|
PY?=python3
|
||||||
|
LEDGER=hledger
|
||||||
|
TOLDG=$(PY) ../toldg.py
|
||||||
|
GETOFX=$(PY) ../getofx.py
|
||||||
|
|
||||||
|
TOLDG_CONFIG=configs/toldg.json
|
||||||
|
GETOFX_CONFIG=configs/getofx.json
|
||||||
|
LEDGER_FILES=$(wildcard processed/*.ldg)
|
||||||
|
OUTPUTDIR=processed
|
||||||
|
LEDGER_ALL=result.ldg
|
||||||
|
|
||||||
|
all: toldg merge
|
||||||
|
|
||||||
|
help:
|
||||||
|
@echo 'Makefile for Ledger automation '
|
||||||
|
@echo ' '
|
||||||
|
@echo 'Usage: '
|
||||||
|
@echo ' make getofx download ofx data and write to CSV '
|
||||||
|
@echo ' make toldg transform CSV files into LDG files '
|
||||||
|
@echo ' make merge merge all ldg files into one '
|
||||||
|
@echo ' make bs show hledger balance '
|
||||||
|
@echo ' make ui open hledger-ui in tree view '
|
||||||
|
@echo ' make clean remove ledger files from output dir'
|
||||||
|
@echo ' '
|
||||||
|
|
||||||
|
getofx:
|
||||||
|
@$(GETOFX) $(GETOFX_CONFIG)
|
||||||
|
|
||||||
|
toldg:
|
||||||
|
@$(TOLDG) $(TOLDG_CONFIG)
|
||||||
|
|
||||||
|
merge:
|
||||||
|
@cat $(LEDGER_FILES) > $(LEDGER_ALL)
|
||||||
|
|
||||||
|
bs:
|
||||||
|
@echo ""
|
||||||
|
@$(LEDGER) bs -V --depth 2 -f $(LEDGER_ALL)
|
||||||
|
|
||||||
|
ui:
|
||||||
|
hledger-ui -V --tree --depth 2 -f $(LEDGER_ALL)
|
||||||
|
|
||||||
|
clean:
|
||||||
|
@[ ! -d $(OUTPUTDIR) ] || rm -rf $(OUTPUTDIR)/*
|
||||||
|
|
|
@ -0,0 +1,28 @@
|
||||||
|
{
|
||||||
|
"secret": "s3cr3t",
|
||||||
|
"client": {
|
||||||
|
"url": "https://ofx.bank.com",
|
||||||
|
"userid": "userid",
|
||||||
|
"org": "B1",
|
||||||
|
"clientuid": "clientuid",
|
||||||
|
"fid": "fid",
|
||||||
|
"bankid": "bankid",
|
||||||
|
"version": 220
|
||||||
|
},
|
||||||
|
"accounts": [
|
||||||
|
{
|
||||||
|
"name": "OFX Checking",
|
||||||
|
"accttype": "checking",
|
||||||
|
"acctid": "111111",
|
||||||
|
"csv_file": "inputs/2018_checking.csv",
|
||||||
|
"fields": [ "", "date", "description", "amount", "", "", "" ]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "OFX Credit",
|
||||||
|
"accttype": "credit",
|
||||||
|
"acctid": "111111",
|
||||||
|
"csv_file": "inputs/2018_credit.csv",
|
||||||
|
"fields": ["date", "", "description", "", "", "amount"]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
|
@ -0,0 +1,17 @@
|
||||||
|
{
|
||||||
|
"input_directory": "inputs",
|
||||||
|
"output_directory": "processed",
|
||||||
|
"mappings_directory": "mappings",
|
||||||
|
"csv_configs": [
|
||||||
|
{
|
||||||
|
"account1": "assets:checking",
|
||||||
|
"file_match_regex": ".*_checking\\.csv",
|
||||||
|
"fields": [ "", "date", "description", "amount", "", "", "" ]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"account1": "liabilities:credit",
|
||||||
|
"file_match_regex": ".*_credit\\.csv",
|
||||||
|
"fields": ["date", "", "description", "", "", "amount"]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
|
@ -0,0 +1,9 @@
|
||||||
|
Details,Posting Date,Description,Amount,Type,Balance,Check or Slip #
|
||||||
|
DEBIT,12/24/2018,"KROGER 12/22",-25.87,DEBIT_CARD,0,,
|
||||||
|
DEBIT,12/20/2018,"KROGER 12/19",-47.77,DEBIT_CARD,0,,
|
||||||
|
DEBIT,12/17/2018,"TARGET",-28.77,DEBIT_CARD,0,,
|
||||||
|
DEBIT,12/04/2018,"TARGET",-45.33,DEBIT_CARD,0,,
|
||||||
|
CREDIT,11/30/2018,"EMPLOYER USA LLC QUICKBOOKS",1337.28,ACH_CREDIT,0,,
|
||||||
|
DEBIT,11/06/2018,"KROGER 11/05",-12.97,DEBIT_CARD,3788.97,,
|
||||||
|
CREDIT,11/02/2018,"EMPLOYER USA LLC QUICKBOOKS",1337.50,ACH_CREDIT,0,,
|
||||||
|
DEBIT,10/30/2018,"Payment to credit card ending in 1337 10/30",-59.90,ACCT_XFER,0,,
|
Can't render this file because it has a wrong number of fields in line 2.
|
|
@ -0,0 +1,6 @@
|
||||||
|
Transaction Date,Post Date,Description,Category,Type,Amount
|
||||||
|
12/03/2018,12/04/2018,UBER ABO13,Travel,Sale,-29.37
|
||||||
|
11/20/2018,11/21/2018,UBER OBC3E,Travel,Sale,-30.41
|
||||||
|
11/19/2018,11/20/2018,KROGER 1337,Groceries,Sale,-10.74
|
||||||
|
11/18/2018,11/19/2018,METRO #83,Groceries,Sale,-12.34
|
||||||
|
10/30/2018,10/30/2018,Payment Thank You - Web,,Payment,59.90
|
|
|
@ -0,0 +1,5 @@
|
||||||
|
|
||||||
|
2018/09/06 Opening Balance
|
||||||
|
assets:checking $-479.86
|
||||||
|
equity:opening balances
|
||||||
|
|
|
@ -0,0 +1,2 @@
|
||||||
|
income:job,EMPLOYER USA LLC QUICKBOOKS
|
||||||
|
assets:transfers:checking-credit,/Payment to credit card ending in 1337/
|
|
|
@ -0,0 +1,4 @@
|
||||||
|
assets:transfers:checking-credit,Payment Thank You - Web
|
||||||
|
expenses:household,TARGET,credit=-28.77;date=2018/12/17
|
||||||
|
expenses:car,METRO #83,credit=-12.34
|
||||||
|
expenses:car,UBER OBC3E,credit=-30.41;date=2018/11/20
|
|
|
@ -0,0 +1,2 @@
|
||||||
|
expenses:groceries,/KROGER/
|
||||||
|
expenses:groceries,TARGET,credit=-45.33;date=2018/12/04
|
|
|
@ -0,0 +1,57 @@
|
||||||
|
|
||||||
|
2018/12/24 KROGER 12/22 ; DEBIT, 12/24/2018, KROGER 12/22, -25.87, DEBIT_CARD, 0, ,
|
||||||
|
expenses:groceries $ 25.87
|
||||||
|
assets:checking $ -25.87
|
||||||
|
|
||||||
|
2018/12/20 KROGER 12/19 ; DEBIT, 12/20/2018, KROGER 12/19, -47.77, DEBIT_CARD, 0, ,
|
||||||
|
expenses:groceries $ 47.77
|
||||||
|
assets:checking $ -47.77
|
||||||
|
|
||||||
|
2018/12/17 TARGET ; DEBIT, 12/17/2018, TARGET, -28.77, DEBIT_CARD, 0, ,
|
||||||
|
expenses:household $ 28.77
|
||||||
|
assets:checking $ -28.77
|
||||||
|
|
||||||
|
2018/12/04 TARGET ; DEBIT, 12/04/2018, TARGET, -45.33, DEBIT_CARD, 0, ,
|
||||||
|
expenses:groceries $ 45.33
|
||||||
|
assets:checking $ -45.33
|
||||||
|
|
||||||
|
2018/11/30 EMPLOYER USA LLC QUICKBOOKS ; CREDIT, 11/30/2018, EMPLOYER USA LLC QUICKBOOKS, 1337.28, ACH_CREDIT, 0, ,
|
||||||
|
income:job $ -1337.28
|
||||||
|
assets:checking $ 1337.28
|
||||||
|
|
||||||
|
2018/11/06 KROGER 11/05 ; DEBIT, 11/06/2018, KROGER 11/05, -12.97, DEBIT_CARD, 3788.97, ,
|
||||||
|
expenses:groceries $ 12.97
|
||||||
|
assets:checking $ -12.97
|
||||||
|
|
||||||
|
2018/11/02 EMPLOYER USA LLC QUICKBOOKS ; CREDIT, 11/02/2018, EMPLOYER USA LLC QUICKBOOKS, 1337.50, ACH_CREDIT, 0, ,
|
||||||
|
income:job $ -1337.50
|
||||||
|
assets:checking $ 1337.50
|
||||||
|
|
||||||
|
2018/10/30 Payment to credit card ending in 1337 10/30 ; DEBIT, 10/30/2018, Payment to credit card ending in 1337 10/30, -59.90, ACCT_XFER, 0, ,
|
||||||
|
assets:transfers:checking-credit $ 59.90
|
||||||
|
assets:checking $ -59.90
|
||||||
|
|
||||||
|
2018/12/03 UBER ABO13 ; 12/03/2018, 12/04/2018, UBER ABO13, Travel, Sale, -29.37
|
||||||
|
expenses $ 29.37
|
||||||
|
liabilities:credit $ -29.37
|
||||||
|
|
||||||
|
2018/11/20 UBER OBC3E ; 11/20/2018, 11/21/2018, UBER OBC3E, Travel, Sale, -30.41
|
||||||
|
expenses:car $ 30.41
|
||||||
|
liabilities:credit $ -30.41
|
||||||
|
|
||||||
|
2018/11/19 KROGER 1337 ; 11/19/2018, 11/20/2018, KROGER 1337, Groceries, Sale, -10.74
|
||||||
|
expenses:groceries $ 10.74
|
||||||
|
liabilities:credit $ -10.74
|
||||||
|
|
||||||
|
2018/11/18 METRO #83 ; 11/18/2018, 11/19/2018, METRO #83, Groceries, Sale, -12.34
|
||||||
|
expenses:car $ 12.34
|
||||||
|
liabilities:credit $ -12.34
|
||||||
|
|
||||||
|
2018/10/30 Payment Thank You - Web ; 10/30/2018, 10/30/2018, Payment Thank You - Web, , Payment, 59.90
|
||||||
|
assets:transfers:checking-credit $ -59.90
|
||||||
|
liabilities:credit $ 59.90
|
||||||
|
|
||||||
|
2018/09/06 Opening Balance
|
||||||
|
assets:checking $-479.86
|
||||||
|
equity:opening balances
|
||||||
|
|
163
toldg.py
163
toldg.py
|
@ -62,7 +62,7 @@ class CsvMapping:
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class LdgTransaction:
|
class Transaction:
|
||||||
"""
|
"""
|
||||||
Class for ledger transaction to render into ldg file.
|
Class for ledger transaction to render into ldg file.
|
||||||
"""
|
"""
|
||||||
|
@ -129,47 +129,64 @@ def get_mappings(mappings_directory: str) -> List[CsvMapping]:
|
||||||
for m in get_mappings_from_file(f)]
|
for m in get_mappings_from_file(f)]
|
||||||
|
|
||||||
|
|
||||||
def get_transactions(csv_file, config: CsvConfig, mappings: List[CsvMapping]):
|
def get_transactions(csv_file: str, config: CsvConfig) -> List[Transaction]:
|
||||||
def date_to_date(date):
|
def date_to_date(date: str) -> str:
|
||||||
d = datetime.datetime.strptime(date, config.input_date_format)
|
d = datetime.datetime.strptime(date, config.input_date_format)
|
||||||
return d.strftime(config.output_date_format)
|
return d.strftime(config.output_date_format)
|
||||||
|
|
||||||
def flip_sign(amount):
|
def flip_sign(amount: str) -> str:
|
||||||
if amount.startswith("-"):
|
return amount[1:] if amount.startswith("-") else "-" + amount
|
||||||
return amount[1:]
|
|
||||||
return "-" + amount
|
|
||||||
|
|
||||||
|
def row_to_transaction(row, fields):
|
||||||
|
""" The user can configure the mapping of CSV fields to the three
|
||||||
|
required fields date, amount and description via the CsvConfig. """
|
||||||
|
t = {field: row[index] for index, field in fields}
|
||||||
|
amount = t['amount']
|
||||||
|
return Transaction(config.currency, flip_sign(amount), amount,
|
||||||
|
date_to_date(t['date']), config.account1,
|
||||||
|
"account2", t['description'], csv_file, ", ".join(row))
|
||||||
|
|
||||||
|
fields = [(i, f) for i, f in enumerate(config.fields) if f]
|
||||||
|
with open(csv_file, 'r') as f:
|
||||||
|
reader = csv.reader(f, delimiter=config.delimiter,
|
||||||
|
quotechar=config.quotechar)
|
||||||
|
for _ in range(config.skip):
|
||||||
|
next(reader)
|
||||||
|
transactions = [row_to_transaction(row, fields)
|
||||||
|
for row in reader if row]
|
||||||
|
return transactions
|
||||||
|
|
||||||
|
|
||||||
|
def apply_mappings(transactions: List[Transaction], mappings: List[CsvMapping]):
|
||||||
def make_equal_len(str_1, str_2):
|
def make_equal_len(str_1, str_2):
|
||||||
max_len = max(len(str_1), len(str_2))
|
max_len = max(len(str_1), len(str_2))
|
||||||
str_1 += " " * (max_len - len(str_1))
|
str_1 += " " * (max_len - len(str_1))
|
||||||
str_2 += " " * (max_len - len(str_2))
|
str_2 += " " * (max_len - len(str_2))
|
||||||
return (str_1, str_2)
|
return (str_1, str_2)
|
||||||
|
|
||||||
def get_account2(transaction):
|
def get_matching_mappings(transaction):
|
||||||
t = transaction
|
t = transaction
|
||||||
matching_mappings = []
|
matching_mappings = []
|
||||||
for mapping in mappings:
|
for mapping in mappings:
|
||||||
pattern = mapping.description_pattern
|
pattern = mapping.description_pattern
|
||||||
if type(pattern) is str and pattern == transaction.description:
|
if type(pattern) is str and pattern != transaction.description:
|
||||||
pass
|
continue
|
||||||
elif type(pattern) is re.Pattern and pattern.match(t.description):
|
elif type(pattern) is re.Pattern and not pattern.match(t.description):
|
||||||
pass
|
|
||||||
else:
|
|
||||||
continue
|
continue
|
||||||
|
|
||||||
specifiers_match = True
|
specifiers_match = True
|
||||||
for attr, value in mapping.specifiers:
|
for attr, value in mapping.specifiers:
|
||||||
if getattr(t, attr) != value:
|
if getattr(t, attr) != value:
|
||||||
specifiers_match = False
|
specifiers_match = False
|
||||||
|
if not specifiers_match:
|
||||||
|
continue
|
||||||
|
matching_mappings.append(mapping)
|
||||||
|
return matching_mappings
|
||||||
|
|
||||||
if specifiers_match:
|
def get_account2(transaction):
|
||||||
matching_mappings.append(mapping)
|
matching_mappings = get_matching_mappings(transaction)
|
||||||
|
|
||||||
if not matching_mappings:
|
if not matching_mappings:
|
||||||
logging.info(f"No match for {transaction}.")
|
logging.info(f"No match for {transaction}.")
|
||||||
e = f"expenses,{t.description},credit={t.credit};date={t.date}\n"
|
return ""
|
||||||
unmatched_expenses.append(e)
|
|
||||||
return "expenses"
|
|
||||||
elif len(matching_mappings) == 1:
|
elif len(matching_mappings) == 1:
|
||||||
return matching_mappings[0].account2
|
return matching_mappings[0].account2
|
||||||
else:
|
else:
|
||||||
|
@ -179,38 +196,23 @@ def get_transactions(csv_file, config: CsvConfig, mappings: List[CsvMapping]):
|
||||||
logging.info(f" {m}")
|
logging.info(f" {m}")
|
||||||
return matching_mappings[0].account2
|
return matching_mappings[0].account2
|
||||||
|
|
||||||
def row_to_transaction(row):
|
|
||||||
t = {field: row[index] for index, field in fields}
|
|
||||||
amount = t['amount']
|
|
||||||
t = LdgTransaction(config.currency, flip_sign(amount), amount,
|
|
||||||
date_to_date(t['date']), config.account1,
|
|
||||||
"", t['description'], csv_file, ", ".join(row))
|
|
||||||
t.account1, t.account2 = make_equal_len(t.account1, get_account2(t))
|
|
||||||
return t
|
|
||||||
|
|
||||||
fields = [(index, field)
|
|
||||||
for index, field in enumerate(config.fields) if field]
|
|
||||||
unmatched_expenses = []
|
unmatched_expenses = []
|
||||||
with open(csv_file, 'r') as f:
|
for t in transactions:
|
||||||
reader = csv.reader(f, delimiter=config.delimiter,
|
account2 = get_account2(t)
|
||||||
quotechar=config.quotechar)
|
if not account2:
|
||||||
[next(reader) for _ in range(config.skip)]
|
unmatched_expenses.append(t)
|
||||||
transactions = [t
|
account2 = "expenses"
|
||||||
for row in reader
|
t.account1, t.account2 = make_equal_len(t.account1, account2)
|
||||||
if row
|
return unmatched_expenses
|
||||||
if (t := row_to_transaction(row))
|
|
||||||
]
|
|
||||||
return transactions, unmatched_expenses
|
|
||||||
|
|
||||||
|
|
||||||
def render_to_file(transactions, csv_file, ledger_file, template_file=""):
|
def render_to_file(transactions: List[Transaction], csv_file: str, ledger_file: str):
|
||||||
content = "".join([LEDGER_TRANSACTION_TEMPLATE.format(t=t)
|
content = "".join([LEDGER_TRANSACTION_TEMPLATE.format(t=t)
|
||||||
for t in transactions])
|
for t in transactions])
|
||||||
|
|
||||||
status = "no change"
|
status = "no change"
|
||||||
if not os.path.isfile(ledger_file):
|
if not os.path.isfile(ledger_file):
|
||||||
with open(ledger_file, 'w') as f:
|
with open(ledger_file, 'w') as f:
|
||||||
f.write(new_content)
|
f.write(content)
|
||||||
status = "new"
|
status = "new"
|
||||||
else:
|
else:
|
||||||
with open(ledger_file, 'r') as f:
|
with open(ledger_file, 'r') as f:
|
||||||
|
@ -223,9 +225,25 @@ def render_to_file(transactions, csv_file, ledger_file, template_file=""):
|
||||||
logging.info(f"{csv_file:30} -> {ledger_file:30} | {status}")
|
logging.info(f"{csv_file:30} -> {ledger_file:30} | {status}")
|
||||||
|
|
||||||
|
|
||||||
def main(config):
|
def write_mappings(unmatched_transactions: List[Transaction], mappings_directory: str):
|
||||||
def file_age(file):
|
""" Write mappings for unmatched expenses for update by the user. """
|
||||||
return time.time() - os.path.getmtime(file)
|
if not unmatched_transactions:
|
||||||
|
return
|
||||||
|
fn = os.path.join(mappings_directory, "unmatched.csv")
|
||||||
|
with open(fn, 'a') as f:
|
||||||
|
writer = csv.writer(f)
|
||||||
|
for t in unmatched_transactions:
|
||||||
|
e = ["expenses", t.description,
|
||||||
|
f"credit={t.credit};date={t.date}"]
|
||||||
|
writer.writerow(e)
|
||||||
|
|
||||||
|
|
||||||
|
def process_csv_file(csv_file, mappings: List[CsvMapping], config: Config):
|
||||||
|
def csv_to_ldg_filename(csv_file: str, config: Config) -> str :
|
||||||
|
r = csv_file
|
||||||
|
r = r.replace(config.input_directory, config.output_directory)
|
||||||
|
r = r.replace(".csv", ".ldg")
|
||||||
|
return r
|
||||||
|
|
||||||
def get_csv_config(csv_file: str, csv_configs: List[CsvConfig]) -> CsvConfig:
|
def get_csv_config(csv_file: str, csv_configs: List[CsvConfig]) -> CsvConfig:
|
||||||
cs = [c for c in csv_configs
|
cs = [c for c in csv_configs
|
||||||
|
@ -236,41 +254,28 @@ def main(config):
|
||||||
raise Exception(f"More than one config for {csv_file=}.")
|
raise Exception(f"More than one config for {csv_file=}.")
|
||||||
return cs[0]
|
return cs[0]
|
||||||
|
|
||||||
def write_unmatched_expenses(unmatched_expenses, mappings_directory):
|
ledger_file = csv_to_ldg_filename(csv_file, config)
|
||||||
if not unmatched_expenses:
|
csv_config = get_csv_config(csv_file, config.csv_configs)
|
||||||
return
|
transactions = get_transactions(csv_file, csv_config)
|
||||||
fn = os.path.join(mappings_directory, "unmatched.csv")
|
unmatched_transactions = apply_mappings(transactions, mappings)
|
||||||
with open(fn, 'a') as f:
|
write_mappings(unmatched_transactions, config.mappings_directory)
|
||||||
for e in unmatched_expenses:
|
render_to_file(transactions, csv_file, ledger_file)
|
||||||
f.write(e)
|
|
||||||
|
|
||||||
def csv_to_ldg_filename(csv_file: str, config: Config):
|
|
||||||
r = csv_file
|
|
||||||
r = r.replace(config.input_directory, config.output_directory)
|
|
||||||
r = r.replace(".csv", ".ldg")
|
|
||||||
return r
|
|
||||||
|
|
||||||
def process_csv_file(csv_file, mappings: List[CsvMapping], config: Config):
|
def process_ldg_file(ldg_file: str, config: Config):
|
||||||
ledger_file = csv_to_ldg_filename(csv_file, config)
|
file_age = lambda file: time.time() - os.path.getmtime(file)
|
||||||
csv_config = get_csv_config(csv_file, config.csv_configs)
|
dest_file = ldg_file.replace(config.input_directory, config.output_directory)
|
||||||
|
status = "no change"
|
||||||
|
if not os.path.isfile(dest_file):
|
||||||
|
status = "new"
|
||||||
|
shutil.copy(ldg_file, dest_file)
|
||||||
|
if file_age(dest_file) > file_age(ldg_file):
|
||||||
|
shutil.copy(ldg_file, dest_file)
|
||||||
|
status = "update"
|
||||||
|
logging.info(f"{ldg_file:30} -> {dest_file:30} | {status}")
|
||||||
|
|
||||||
transactions, unmatched = get_transactions(
|
|
||||||
csv_file, csv_config, mappings)
|
|
||||||
write_unmatched_expenses(unmatched, config.mappings_directory)
|
|
||||||
render_to_file(transactions, csv_file, ledger_file)
|
|
||||||
|
|
||||||
def process_ldg_file(ldg_file: str, config: Config):
|
|
||||||
dest_file = ldg_file.replace(
|
|
||||||
config.input_directory, config.output_directory)
|
|
||||||
status = "no change"
|
|
||||||
if not os.path.isfile(dest_file):
|
|
||||||
status = "new"
|
|
||||||
shutil.copy(ldg_file, dest_file)
|
|
||||||
if file_age(dest_file) > file_age(ldg_file):
|
|
||||||
shutil.copy(ldg_file, dest_file)
|
|
||||||
status = "update"
|
|
||||||
logging.info(f"{ldg_file:30} -> {dest_file:30} | {status}")
|
|
||||||
|
|
||||||
|
def main(config):
|
||||||
input_files = get_files(config.input_directory)
|
input_files = get_files(config.input_directory)
|
||||||
config.csv_configs = [CsvConfig(**c) for c in config.csv_configs]
|
config.csv_configs = [CsvConfig(**c) for c in config.csv_configs]
|
||||||
mappings = get_mappings(config.mappings_directory)
|
mappings = get_mappings(config.mappings_directory)
|
||||||
|
@ -286,7 +291,7 @@ def main(config):
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
logging.basicConfig(stream=sys.stdout,
|
logging.basicConfig(stream=sys.stdout,
|
||||||
level=logging.DEBUG,
|
level=logging.INFO,
|
||||||
format='%(message)s')
|
format='%(message)s')
|
||||||
try:
|
try:
|
||||||
config_file = sys.argv[1]
|
config_file = sys.argv[1]
|
||||||
|
|
Loading…
Reference in New Issue